MetaDesigner: Advancing Artistic Typography through AI-Driven, User-Centric, and Multilingual WordArt Synthesis

Response to the Weakness:

1. We appreciate the reviewer's insightful comment regarding the potential complexity of our multi-agent system and iterative optimization process for non-technical users. We agree that user-friendliness is crucial. Therefore, We have developed a user-friendly interface that abstracts away the underlying complexity of the system. Users can interact with the system through simple prompts and intuitive controls, without needing to understand the technical details.

2. We have listed efficiency, please refer to Review #2-Q3, all the experiments are conducted on a Tesla V 100 GPU.

Response to the Questions：

1）We have incorporated additional comparison by separating Glyphs and Texture into the current version of the manuscript, please refer to Figure 18.

2）Please refer to Figure 21.

3）Our system does have some failure cases. a) Some images contain redundant text content. As pointed out by Reviewer #2 (zqVY) with the example of `Renaissance' (Figure 8, lower right), this is due to the low quality of the LoRA model used by Texture Designer, which also highlights the importance of enhancing the quality of the LoRA models employed by Texture Designer. b) The readability of text in some images is insufficient (Figure 5, the last two examples in the 'Ours' row may exhibit this issue). To enhance the text's and background's harmony, we utilize a high control scale, while acknowledging that this may marginally impact readability in some instances. This highlights the need for automated systems capable of quantitatively evaluating both artistic and textual qualities in generated outputs. Our subsequent research will be directed towards developing such systems.

Thank you for your additional feedback! We will continue to refine the manuscript before the deadline, adding more details and examples to improve reader comprehension.

Response to the Weakness:

1. Thank you for your valuable feedback. While the construction of the style LoRAs library is indeed a significant component of our approach, it’s the novel combination of this technique with other innovative modules that drives the substantial improvements in "Accuracy and Creativity." Our primary goal is to empower non-professional users to create high-quality, creative WordArt with minimal effort. To achieve this, we introduce a multi-agent system that leverages the strengths of various techniques. Our proposed framework introduces several novel aspects:

• Enhanced Prompt Engineering: The Pipeline Agent leverages a large language model to extend user prompts into detailed descriptions, enabling more precise control over the generated WordArt.

• Semantic-Aware Glyph Synthesis: The Glyph Agent introduces semantic deformable capabilities, allowing for flexible glyph transformations and improving the overall aesthetic appeal of the generated WordArt.

• Adaptive Texture Rendering: The Texture Agent employs a ToT (Tree-of-Though) selection scheme to dynamically choose the most suitable LoRA, ensuring alignment with user preferences and generating diverse and high-quality textures.

• Iterative Refinement with Feedback: The feedback mechanism allows for interactive refinement of the generated WordArt, enabling users to provide feedback and guide the synthesis process toward their desired outcome.

By combining these techniques, our approach offers a comprehensive solution for generating creative and high-quality WordArt, even for non-professional users. We believe that our work pushes the boundaries of WordArt synthesis and provides a valuable contribution to the field. We will update the introduction and the details to highlight our contribution. We will update the manuscript to highlight the novelty.

2. Thank you for your valuable feedback. We have incorporated additional evaluation feedback studies and examples into the current uploaded version of the manuscript. The effect of the evaluation is updated in Table 1, and in Table 2; the VLMs evaluation results are listed in Table 3. More exemplary case is listed in Figure 17

Response to the Questions:

1. We have incorporated additional intermediate results and analyses in the full pipeline into the final version of the manuscript, which will be uploaded shortly.

2. "Texture Designer" does not involve training any additional parameters. Instead, we curated a library of 68 pre-trained LoRA weights, which are loaded at inference time to adapt the model to specific user requests. To efficiently select the optimal LoRA, we implemented a Tree-of-Thought strategy. This approach leverages user-provided prompts (and their extended versions generated by the Pipeline Agent) to identify the most relevant LoRA weights. The entire LoRA library, originally 34GB in size, can be compressed to less than 17GB using fp16, enabling seamless loading into memory. The negligible switching time (~ 0.15s) between pre-loaded LoRAs enables efficient processing of four images within 8-9 seconds.

3. Information about the inference efficiency}}: A comparison of image generation time and computational resources required, relative to other models:

The MetaDesigner excels in inference efficiency, especially for single-round scenarios. The PipelineDesigner and evaluator complete their tasks in a mere 1$\sim$3 seconds, while the GlyphDesigner and TextureDesigner require approximately 26 and 10 seconds, respectively. Even when considering 3-5 rounds of iteration for high-quality WordArt synthesis, the total time cost is still under 2-4 minutes, surpassing the efficiency of comparison methods.

4. a) To comprehensively assess the GPT-4/GPT-4V model on TextureDesigner, we utilize 25 test prompts, organized into 5 distinct thematic categories. The ToT-Selection results are listed in Appendix D. The "World Peace" and "Colorful World" themes are more open-ended compared to "Future Technology." GPT-4V exhibits more diverse responses on the open-ended themes, suggesting a better understanding of human intent. However, for the more constrained "Future Technology" theme, GPT-4V is more consistent than GPT-4. b) Comparisons are illustrated in Figure 19. GPT-4V exhibits a more accurate and nuanced ability to describe visual content compared to LLava. As demonstrated in our initial experiment, LLava occasionally produces "hallucinations" in its lighting descriptions, particularly when evaluating consistency with the given prompt. In contrast, GPT-4V accurately identifies discrepancies between the described lighting conditions and the actual visual content. Consequently, GPT-4V often assigns lower scores to images compared to LLava.

5. a) Refer to Reviewer #3 Q3. b) The phenomenon where "Renaissance" is generated as part of the text is attributed to the LoRA model, which may have been trained on a dataset containing images with the word `Renaissance.' This occurrence is not a universal phenomenon in our experiments (for more examples, see Figure 13), and it certainly highlights the importance of higher-quality LoRA models.

[1] Word-As-Image for Semantic Typography. ACM Trans. Graph. 42(4): 151:1-151:11 (2023)

[2] DS-Fusion: Artistic Typography via Discriminated and Stylized Diffusion. ICCV 2023: 374-384

[3] WordArt Designer: User-Driven Artistic Typography Synthesis using Large Language Models. EMNLP 2023: 223-232

Thank you for your additional feedback!

1. Please refer to the Figure 20 for the details of each step. The Glyph image generated by the "GlyphDesigner" is illustrated in Figure 23, and the effect of the different fonts for glyph transformation is illustrated in Figure 22.

2. Model Information and Access: We've compiled information about the models used in this project into a file (model_info.json) accessible at Anonymous Link. These models leverage the power of the open-source community; they were downloaded from Civitai and Libib, where you'll find detailed introductions by their authors. For our purposes, we've categorized the models by style (see Figure 11) and selected them based on Tree-of-Though (ToT).

   Model Characteristics: LoRA models typically have 10+ million to 100+ million parameters, depending on the LoRA rank (typically 32-128). A small portion of our library includes standard Stable Diffusion (SD) base models. Thanks to safetensors, switching between base models is very fast (around 0.5 seconds). LoRA models themselves typically require dozens to hundreds of images for training. Community-developed base models are often derived by combining and improving existing models, rather than starting from scratch.

3. We fully concur with your perspective: creativity and readability are not inherently conflicting in the design. We would like to offer a slight clarification regarding our response to Reviewer 9F4p Q3, where we intended to convey that the integration of text with the background and the readability of the text itself requires a balance. That is to say, when we aim for a more harmonious integration of text with the background, we may sacrifice some readability, and conversely, when we enhance the readability of the text, the degree of integration with the background may decrease. We apologize for the initial imprecise use of the term `creativity'.

   Regarding the enhancement of readability, we have conducted some explorations, but we have found that there is a significant inaccuracy in the automated assessment of the readability of artistic fonts. Even the most optimal Optical Character Recognition (OCR) methods currently available, such as PP-OCRv4, have an extremely low recognition rate for artistic fonts (10%<), whereas humans can effectively identify artistic fonts by integrating contextual information. Consequently, we are unable to construct an automated pipeline to accurately perceive the objective impact of variations in glyph shapes and the extent of texture rendering on readability.

   At the current stage, achieving precise control over the degree of artistic refinement to ensure that both creativity (artistic quality) and readability are maintained at a high level requires human intervention in judging various combinations of changes. This makes the design process more complex and time-consuming, which goes against the original intention of our system's ease of use. Therefore, how to ensure that the generated results possess a high level of both creativity (artistic quality) and readability is a highly valuable question. The core lies in the high-precision automated perception of artistic quality and readability, which is also the central direction of our future research.

We will continue to refine the manuscript before the deadline, adding more details and examples to improve reader comprehension.

Response to the Questions:

1. We aim to achieve a dual objective: synthesizing an image with the highest evaluation score while simultaneously identifying the optimal hyperparameters$P^{gly}$ and $P^{tex}$. To this end, we employ the argmax function to optimize the hyperparameter settings that maximize the evaluation score.

2. Optimizing Prompts .

f"""As a prompt optimization expert, please improve the
    following prompt based on the evaluation results.
        Original Prompt:
        "{original_prompt}"

        Evaluation Results:
        Consistency Score: {evaluation['consistency']['score']}
        Reason: {evaluation['consistency']['reason']}

        Please optimize the prompt following these rules:
        - Duplicate important nouns
        - Add emphasis using () brackets for key elements
        - Add specific spatial relationships
"""

f"""As a prompt optimization expert, please improve the
    following prompt based on the evaluation results.
        Original Prompt:
        "{original_prompt}"

        Evaluation Results:
        Style Score: {evaluation['style']['score']}
        Reason: {evaluation['style']['reason']}

        Please optimize the prompt following these rules:
        For low style scores, extract the key style elements from prompt & reseaon
        - Style: Anime, Cartoon, Realistic, Cyberpunk, Cybernetic, Cybernetic, Cybernetic
        - Enhance style-related descriptors, .
        - Add specific artistic technique references
        - Strengthen atmosphere descriptions
"""

f"""As a prompt optimization expert, please improve the
    following prompt based on the evaluation results.
        Original Prompt:
        "{original_prompt}"

        Evaluation Results:
        Quality Score: {evaluation['quality']['score']}
        Reason: {evaluation['quality']['reason']}

        Please optimize the prompt following these rules:
        For low quality scores, add technical specifications/keywords following the prompt & reason  
        - Resolution: 8K, 4K, High resolution
        - Details: Highly detailed, Intricate, Minute details
        - Lighting: 4K lightning, Natural light, Artificial light, Backlit, Spotlight, Ambient light
"""

f"""As a prompt optimization expert, please improve the
    following prompt based on the evaluation results.
        Original Prompt:
        "{original_prompt}"

        Evaluation Results:
        Creativity Score: {evaluation['creativity']['score']}
        Reason: {evaluation['creativity']['reason']}

        Please optimize the prompt following these rules:
        For low creativity scores:
        
        - Reset the parameters from the original prompt and reason:
            - Glyph Parameters
                - FontTyle as follow ,semantic is more creativity: 
                    Formal
                        - Formal, 
                        - Elegant, 
                        - Classic, 
                        - Traditional
                    Semantic  

                - Iterations: 80~180, the higher the more artistic
            - update the texture prompt
                - Add unique viewpoints
                - Include unexpected elements
                - Enhance artistic innovation aspects
"""

All the feedbacks are strings that will be further parsing by the GlyphDesigner and TextureDesigner.

3. "Text accuracy" is used to reflect whether the text in the images is written correctly. Specifically, during the user study process, we present users with six images (corresponding to the six methods in Table 1) and the input text, then prompt them with the question: "Select the images where you believe the text is written correctly (multiple choices allowed)." The percentages in Table 1 represent the proportion of times each method was chosen by the users. It is worth noting that the "Text accuracy" question allows multiple selections, hence the sum of all values in the second row of Table 1 does not equal 100. In contrast, the "Aesthetics & Creativity" question is a single-choice question, so the sum of all values in the third row of Table 1 exactly equals 100.

Response to the Weakness:

1. Thank you for your valuable feedback. $s^{gly}$ represents the extended prompt by the Pipeline Agent,

• Regarding normal and traditional glyphs, we categorized glyphs based on their stylistic attributes. "Normal glyphs" refer to modern fonts commonly used in printed texts, such as Song and Hei typefaces. In contrast, "traditional glyphs" encompass Chinese calligraphy styles like Li, Xing, and Cao scripts, characterized by their historical and artistic significance. To illustrate the distinction, we will provide visual examples in the manuscript.

• For Semantic Glyph Transformation, the Glyph Agent employs differential rendering techniques to modify target glyphs [1,2]. The Glyph Designer deforms the glyph based on the semantic concept provided by the pipeline designer. We utilize both score diffusion loss and CLIP loss to guide the deformation process, ensuring that the glyph evolves toward the desired semantic concept. This optimization process typically requires 80 -180 iterations (can be set by the user/agent), depending on the desired level of deformation. We set to differential rendering size to 350x350 to reduce the time cost. The loss function is a score diffusion loss, which ensures that the generated glyphs maintain content consistency while aligning with the desired semantic concept.

In Section 3.4, $\mathcal{L}_{u}$ and $\mathcal{L}_{m}$ represent the objective loss (average score) from the user and LLaVA model feedback respectively.

2. Thanks for your valuable feedback, we follow your suggestion to conduct further studies:

• GPT-Augmented SDXL: Actually, for all methods in Figure 5, we employed the same prompt. Following your suggestion, we have added GPT-Augmented SDXL to the manuscript and correspondingly included the visualization results in the SOTA comparison, denoted as `SDXL-Augmented/Aug.'. Our experiments revealed that while the more detailed prompts provided by GPT augmentation can enhance image quality, they do not significantly improve the correctness of SDXL's text generation. We surmise that this is primarily due to the inherent limitations in SDXL's comprehension of text.

• Divide "Aesthetics & Creativity" into two dimensions: We have divided "Aesthetics & Creativity" into two distinct dimensions and conducted a new user study. The results are presented in the manuscripts.

• Regarding the discussions about SSIM and LPIPS in Table 2, While the SSIM and LPIPS metrics can partially reflect the spatial structural similarity between the generated images and the ground truth images, they do not fully account for the readability of the text. We have also attempted to employ OCR to assess readability. However, we found even the most advanced OCR methods, such as PP-OCRv4, to have a low recognition rate for the artistic fonts generated by our approach as well as other methods, rendering it an unreliable means of evaluation. Consequently, we have introduced the evaluation dimension of "Text Accuracy" in our user study to manually assess the readability of the text in the generated images. We will elucidate the meaning of `Text Accuracy' in our subsequent responses.

• Results of the proposed method, without a relative agent: We have included "ours (w/o agents)" in both the user study and numerical experiments. For details, please refer to our manuscript. Additionally, we have included the corresponding visualizations. We have also attempted to employ OCR to assess readability. However, we found even the most advanced OCR methods, such as PP-OCRv4, to have a low recognition rate for the artistic fonts generated by our approach as well as other methods, rendering it an unreliable means of evaluation. Consequently, we have introduced the evaluation dimension of "Text Accuracy" in our user study to manually assess the readability of the text in the generated images. We will elucidate the meaning of "Text Accuracy" in our subsequent responses.

3. Thanks for your valuable feedback. The creativity and the readability of the text indeed require a balance. If users prioritize ensuring readability, this can be achieved through the following two methods: 1) In Glyph Designer, by reducing the deformed stroke areas to retain more of the original stroke information; 2) In Texture Designer, by increasing the control scale of the ControlNet to enhance the clarity of the text edges, thereby improving readability. This also incorporates user feedback to align with user preferences.

[1] Word-As-Image for Semantic Typography. ACM Trans. Graph. 42(4): 151:1-151:11 (2023)

[2] WordArt Designer: User-Driven Artistic Typography Synthesis using Large Language Models. EMNLP 2023: 223-232