Text-Aware Real-World Image Super-Resolution via Diffusion Model with Joint Segmentation Decoders

We are very grateful for the reviewer's encouraging and positive assessment of our work, especially regarding its performance, novelty, and the data synthesis pipeline. The detailed and insightful feedback has been invaluable in helping us to strengthen our paper. We address each concern point-by-point below.

Q1: The model's performance on more severe degradation and larger upscaling factors (e.g., ×6 or ×8).

A1: We thank the reviewer for the question. We respectfully clarify that up to the ×4 super-resolution setting is a common and established practice in recent state-of-the-art literature for both general real-world SR and text image SR. For instance, methods we compared against, such as HAT, SeeSR, OSEDiff, MARCONet, and DiffTSR, as well as other recent works like CATANet [1] and PFT-SR [2], all present their primary results at a maximum of ×4 magnification. This is because at higher scaling factors, severe information loss in the input makes it exceedingly difficult for any model to reliably reconstruct faithful structures and local details, a challenge that is particularly acute for the intricate structures of text. We chose the ×4 setting in our main paper to ensure a fair and direct comparison with these contemporary methods.

Furthermore, we believe it is important to consider the reliability of results on inputs that are completely illegible to humans. While a model might occasionally generate a correct character from such an input, this result is likely due to statistical guessing or overfitting rather than genuine reconstruction. This may appear impressive but is not always reliable or beneficial for practical applications. For example, as shown in our paper, MARCONet incorrectly reconstructs “清华大学” (Tsinghua University) as “国华大学” (Guohua University) and, in the supplementary material, misinterprets "Huancheng" as "Wuancheng". Such semantic errors are often less acceptable in real-world use cases than a blurry but structurally correct result.

Nevertheless, to address the reviewer's concern, we have conducted additional experiments for the ×8 setting. Since there is a lack of paired real-world text SR datasets for such a high scaling factor (Real-CE only provides up to ×4), we simulated an ×8 scenario by downsampling the 13mm focal length images from Real-CE by an additional factor of 2. We then fine-tuned both OSEDiff and our method on this new ×8 dataset. The results are as follows:

Our method continues to outperform the baseline, demonstrating its robustness even under more severe degradation. In the next version of our paper, we will introduce evaluations and visual comparisons for more methods on higher-scale settings.

Q2: The necessity of the specific word "text" and the impact of alternative prompts.

A2: We thank the reviewer for raising this important point. As requested, we conducted experiments using different keywords in the prompt:

In fact, the specific keyword is not the most critical aspect. The word "text" can be viewed as analogous to a "trigger word" in LoRA or a "pseudo-word" in Textual Inversion. Its primary function is to activate the specialized knowledge the model acquired during fine-tuning. We chose "text" because it is a more general and broadly applicable term for text-related content compared to "sign," "label," or "writing."

We also plan to explore more advanced prompting strategies in future work, such as embedding the specific textual content or dynamic background descriptions into the prompt to further enhance quality. However, this remains a significant challenge for current diffusion foundation models, as none can yet reliably generate long, coherent strings of text from a prompt, which is why we adopted the current, more robust prompting strategy.

Regarding prompt-free inference, this scenario is effectively covered by the ablation study in Q3, "disabling the Text-Aware Cross-Attention mechanism". In that setting, the cross-attention response to the text prompt is completely removed from the text segmentation estimation process. This eliminates the model's explicit text-awareness, leading to weaker quantitative metrics and visibly blurrier text regions, demonstrating that the text preservation capability is not activated.

Q3: Clarification on the "disabling the Text-Aware Cross-Attention mechanism" ablation study.

A3: We apologize for the lack of clarity in our initial description. In the "disabling the Text-Aware Cross-Attention mechanism" (w/o TACA) setting, we replace the aggregated text-aware cross-attention feature, $**a**_{tex}$, with the denoised image latent feature, $\hat{**z**}_H$, as the input to the Text Segmentation Decoder. This is done to validate the effectiveness of using the specific text-aware cross-attention signal.

As shown in Table 2 and Figure 5 of our paper, disabling TACA leads to a significant drop in TADiSR's performance, evidenced by weaker numerical scores and blurrier text regions. We will ensure this detailed explanation of the ablation settings is included in the revised version of our paper.

Q4: The OCR-A score gain (Δ) over the raw LR inputs.

A4: Thank you for this excellent suggestion. This is a very reasonable way to evaluate performance. We report the OCR-A gain on the Real-CE-val dataset below:

Notably, all prior methods exhibit a decrease in text legibility after super-resolution. This is understandable for general-purpose SR models that lack text awareness. For specialized text SR models like MARCONet and DiffTSR, the performance drop occurs for two main reasons: 1) they can mis-recognize a character, causing a correctly identified character in the LR input to become incorrect, and 2) these patch-based methods struggle with long or vertically-oriented text, leading to performance degradation in real-world scenes. As described in Line 247 of our paper, we already provided these models with text regions pre-cropped by an OCR model; otherwise, they would fail to produce any valid output on full images.

In contrast, our method is the only one to achieve a positive gain, improving the OCR accuracy by 10.7% over the raw input. This underscores the significance of our full-image, text-aware super-resolution approach.

Q5: Overlaying recognized text strings on the qualitative comparison figures.

A5: Thank you for this constructive suggestion. Due to the rebuttal policy, we are not permitted to upload new images. Therefore, we present the OCR results in a formatted table below, which we hope you can reference alongside the figures in our main paper. We used the same multilingual PP-OCR V4 model as in our main experiments. All characters that differ from the Ground Truth (GT) are marked in bold, and each missing character is represented by an underline.

Figure 3 OCR Results:

Figure 4 OCR Results:

Thanks for the diligent review and positive feedback on the novelty and design of our work. We appreciate the insightful questions, which help us improve the clarity of our paper. We address each of your points below.

Q1: The name of the joint seg decoder is confusing.

A1: Thank you for pointing this out. As mentioned on Line 38 of the main text, "Joint Segmentation Decoders" is an abbreviation for "joint image–text segmentation decoder." To eliminate any potential ambiguity and make this clearer, we will revise the figure to explicitly label the two parallel branches as the "Image Decoder" and the "Text Segmentation Decoder." We apologize for any confusion this may have caused.

Q2: There is no explanation about the orange squares, which could be the LoRA?

A2: Your assumption is correct; the orange squares indeed represent the LoRA adapters. In Figure 2, we have indicated this with the label "LoRA Adapters" positioned above the U-Net architecture. We adopted this visualization style as it is a common convention, similar to the approach used in prior works such as Image2Image Turbo [1]. We will ensure this label is more prominent in the revised version to avoid any misunderstanding.

Q3: Why does the $c_{tex}$ need to be located? Isn't it the last one in the $c_y$?

A3: Your understanding is correct for our current implementation. The term "locating" refers to the process of acquiring the specific cross-attention map corresponding to the "text" token. In our current setup with a fixed prompt, this is indeed achieved by simply accessing the token at a known, fixed index.

However, we used the more general term "locating" to maintain broader applicability for future work. This phrasing accommodates potential extensions to dynamic prompts, where the position of the "text" token might vary and would require an explicit search or location step. This choice of terminology was intended to keep the description general rather than tying it to our specific, fixed-prompt implementation.

Q4: Why are $a_{tex}$ concatenated along the channel dimension and how can it be used to generate the segmentation map solely without $z_H$?

A4: Thank you for this insightful and detailed question. It has two parts, which we will address in order.

1. Why Concatenate? As we mentioned in our preliminary experiments (Line 28), techniques like DAAM have shown that aggregated cross-attention maps hold strong spatial semantic information that can be guided to focus on specific regions (in our case, text). DAAM typically achieves this by upsampling and summing the maps. We reasoned that concatenating these maps along the channel dimension, rather than summing them, would be a more lossless approach. This method preserves the distinct information from each attention layer, creating a rich feature representation that serves as the initial latent input for the text segmentation task.
2. How it works with $z_H$: Crucially, the segmentation map is not generated solely from these attention maps. This addresses the second part of your question. Our architecture facilitates interaction with the image latent $z_H$. This is achieved through our proposed Cross-Decoder Interaction Blocks (CDIBs). As detailed in the paper, CDIBs are dual-stream blocks that enable the exchange of features between the image decoding branch (processing $z_H$) and the text segmentation branch (processing $a_{tex}$). This interaction allows the segmentation decoder to be informed by the fine-grained spatial details from $z_H$, leading to a precise and contextually aware final segmentation map.

Q5: It would be better to discuss the parameters, FLOPs, and inference time compared to other methods by introducing an additional VAE decoder.

A5: Thanks for the suggestion. We provide the table below containing a comparative analysis against standard diffusion model adaptation techniques. All metrics were measured on a single NVIDIA H20 GPU with an input resolution of 512x512 pixels.

Here is a breakdown of the comparison:

• Comparison with LoRA: Our method introduces approximately 35.68M extra trainable parameters beyond a standard LoRA setup, primarily from the new segmentation decoder and the Cross-Decoder Interaction Blocks (CDIBs). While this is a modest increase in parameters, the additional FLOPs and a slight increase in inference time are a direct result of the parallel decoding process. We believe this is a very reasonable trade-off for the substantial improvement in text fidelity, a capability that standard LoRA fine-tuning alone cannot provide.
• Comparison with ControlNet: Our approach is significantly more parameter-efficient. A standard ControlNet adds over a billion parameters by duplicating the U-Net encoder blocks. In contrast, our method achieves fine-grained structural control over text regions with only a fraction of the parameters (~3.8% of a standard ControlNet) and a manageable increase in inference latency. This highlights the efficiency of our joint decoding design for the specific task of text-aware super-resolution.

In summary, while the multi-scale feature interactions within our Joint Segmentation Decoders introduce a certain computational overhead, the resulting increase in latency is acceptable. The reason a large model like ControlNet can be relatively fast is that it operates on a single scale within the latent space. However, this single-scale manipulation is often insufficient for restoring heavily degraded text details, which demands the multi-scale, fine-grained feature processing that our architecture provides.

As this is one of the first works to tackle full-image text super-resolution, our primary focus has been on maximizing model performance and restoration quality. We acknowledge that for practical deployment, further optimization techniques could be applied. We will explore these avenues and discuss them in detail in the next version of our work.

[1] One-Step Image Translation with Text-to-Image Models.

We sincerely thank the reviewer for their insightful comments and high praise for our model's advantages, particularly its visual results and novelty. We will now address each of your questions in detail.

Q1: It remains unclear whether the authors have made any attempts to mitigate the domain gap between the synthetic and real-world data pairs.

A1: Thank you for this insightful question. We have provided a detailed analysis on this point in our response to Reviewer oXY4, Question 3. Due to space limitations, we kindly ask you to refer to that response for a comprehensive explanation. We appreciate your understanding.

Q2: The super-resolution of text samples may introduce distortions into the text regions. Moreover, the text segmentation masks may no longer perfectly align with the super-resolved text patches.

A2: Thank you for your meticulous observation. Our data pipeline in the main paper needs to be improved. It is crucial to note that most publicly available Chinese text datasets are designed for text recognition, and their image quality is often inconsistent, containing noise, artifacts, and other degradations. Training directly on such data would teach the model to ignore these degradations rather than correct them.

To prevent the Real-SR method from introducing distortions, we implemented two key quality control measures during data synthesis:

1. Upscaling Before Super-Resolution: We first enlarge the scale of the text image patches before applying the super-resolution model. This ensures that the structure of the text, which now occupies a larger portion of the patch, is better preserved during the SR process.
2. OCR-based Filtering: To avoid distortions in smaller text, which can negatively impact training, we use an OCR detector to extract text from both the original and the super-resolved patches. We only include samples in our training set where the edit distance between the two versions is minimal. This process effectively guarantees the quality of the text regions.

We agree that the Super-Resolution step should indeed precede Text Segmentation Prediction to ensure perfect alignment between the final image content and the segmentation masks. As we are unable to upload images during this rebuttal phase, we have provided a corrected text-based flowchart of our data synthesis pipeline (Please refer to R1A2 due to the character limitation) and will include a properly rendered diagram in the next version of the paper.

Q3: Please provide a detailed analysis of the impact of the dual-decoder design on inference speed.

A3: Thanks for the suggestion. We provide the analysis in A2Q5 due to the character limitation.

Q4: Could the authors evaluate whether training with FTSR improves performance on Real-CE-val?

A4: To demonstrate the effectiveness of our proposed FTSR dataset, we have conducted an ablation study comparing the performance of both our model and the OSEDiff baseline on the Real-CE validation set, with and without pre-training on FTSR, as shown in the following table:

As the results show, pre-training on FTSR improves the performance of both models on the real-world Real-CE dataset across various metrics. This underscores the necessity of incorporating large-scale synthetic data and validates the effectiveness of our data synthesis strategy.

Q5: Whether other baseline methods benefit from using FTSR to demonstrate its general utility. Could the improved performance be partly due to exposure to Chinese characters during training, which competing methods lack?

A5: In our submitted version, for diffusion-based models with publicly available training code, such as SeeSR and OSEDiff, we followed the exact same pre-training and fine-tuning procedure as our own method (as described in Line 245 of the main paper). This includes pre-training on our synthetic FTSR dataset.

Furthermore, the table in A4 reveals another important insight: while baseline models without text-aware capabilities do benefit from FTSR, their performance gain—especially on text-related metrics like OCR-A—is significantly smaller than ours. For instance, OSEDiff's OCR-A improvement from FTSR is only 12.4% of the improvement seen by our model. This disparity strongly suggests that the performance gap is not merely due to data exposure but is fundamentally driven by our model's architecture. The superior ability of our model to leverage the FTSR dataset highlights the importance of its text-aware design and validates the rationality and efficiency of our architectural choices for full-image text super-resolution.

Q6: It is suggested to provide more examples on other languages in addition to Chinese to show its overall ability.

A6: Thank you for the suggestion. In fact, our experimental setup is already bilingual. Both the CTR dataset (used for synthesizing the training set) and the Real-CE test set contain bilingual (Chinese and English) samples. Examples of this can be seen in Figure 1 and Figure 4 of the main paper, with additional English-centric samples in Figure 8 and Figure 9 of the supplementary material.

It is worth noting that the reconstruction of English text is significantly less challenging than that of languages with complex stroke-based characters like Chinese, due to the limited number of letters in the English alphabet. A simple fine-tuning of a base model (like SDXL or SD3) can achieve considerable performance in English-only scenarios. Our research was motivated by the significant lack of attention in the academic community to super-resolution tasks for these more complex languages.

Given that we cannot provide additional images during the rebuttal phase, we will include more performance comparisons in English-rich text scenarios (e.g., on a dataset like TeleScope) in the next version of our paper to offer readers a more comprehensive view.

Q7: Incorporating additional metrics such as OCR-based F1 scores or other metrics would provide a more comprehensive and robust assessment of text fidelity.

A7: We appreciate this constructive feedback. In the experiments in our main paper, we initially chose to perform text detection on the ground-truth (GT) images first and then use those bounding boxes to evaluate recognition on the predicted images. This was done to isolate the text recognition capability from potential errors in the OCR model's text detection stage. However, as the reviewer correctly implies, this approach causes the Precision, Recall, and F1-score to converge to the same value, which is why we did not report them separately.

In response to the reviewer’s suggestion, we have now conducted a full, standard OCR evaluation for every method. This process includes both text detection and text recognition on the output images, where predicted boxes are matched to GT boxes based on IoU and text similarity thresholds. The resulting Precision, Recall, and F1-Score are reported below:

As can be seen, our method remains state-of-the-art across all three metrics, demonstrating its robust performance advantage in a more comprehensive evaluation setting.

Q8: It would be helpful to include a table comparing inference speed, memory usage, and model size across all methods.

A8: Thank you for this practical suggestion. We have compiled a table comparing the efficiency of our model against several representative methods cited in our paper, including GAN-based and one-step diffusion-based methods. All benchmarks were conducted on an NVIDIA H20 GPU with an input resolution of 512x512 pixels.

As the results show, our method demonstrates a highly competitive profile in terms of trainable parameters, FLOPs, and inference time, especially considering the state-of-the-art performance it achieves in full-image text super-resolution. While one-step models like OSEDiff are faster, our approach maintains a practical inference speed with a remarkably small number of trainable parameters, highlighting its efficiency.

Thanks for your diligent review and the positive feedback on the innovation and performance of our work. We appreciate the opportunity to address the reviewer's questions and provide further clarification as below.

Q1: The ablation study should cover the full model. The impact of CDIB should be studied. The authors should validate the effectiveness of the pipeline itself.

A1: Thank you for the suggestion. To maintain the clarity of the main paper, our initial ablation study focused on primary contributions. As requested, we have conducted additional experiments targeting the Cross-Decoder Interaction Block (CDIB) to provide a more comprehensive analysis.

First, we present the quantitative results for these new ablation settings on the FTSR-TE dataset below:

Below, we detail the setup and observations for each experiment:

Experiment 1: Disabling Feature Interaction within CDIB

• Setup: In this experiment, we kept the CDIB modules in the architecture but disabled the cross-decoder feature exchange mechanism within them. Each branch (image and segmentation) of the CDIB processed its features independently without receiving features from the other.
• Observation: This led to a noticeable degradation in text structure fidelity. While the model could still perform both tasks in parallel, the synergy was lost. As the table shows, all metrics declined compared to our full model. The resulting segmentation masks were less accurate, and text in the super-resolved output appeared blurrier. The OCR-A score dropped significantly, confirming that the interactive feature fusion within CDIB is crucial for the mutual enhancement of the two decoders.

Experiment 2: Removing the CDIB Modules Entirely

• Setup: We removed the CDIB modules from the architecture altogether. The image and segmentation decoders operated as two completely separate, parallel pathways from the latent code, with no points of interaction.
• Observation: This configuration performed worse than Experiment 1 and approached the performance of the "w/o JSD" variant from our main paper. Without any architectural connection to facilitate information sharing, the model struggled to leverage segmentation priors to guide text reconstruction effectively. This result underscores the necessity of the CDIB module itself as the structural bridge enabling our joint decoding framework to succeed.

Q2: The pipeline section lacks a flowchart.

A2: Thank you for this valuable suggestion. A flowchart would significantly improve the clarity of the data synthesis pipeline. As it is not feasible to add graphical figures during the rebuttal phase, we have prepared a text-based flowchart below.

[START]
   |
   v

 1. Source Text: CTR Dataset
    (Text Image Patches with variable quality)

   |
   v

 2. Super-Resolution via Real-SR Model
    * Note: To preserve structure, patches are upscaled BEFORE this
      SR step.

   |
   v

 3. Quality Control: OCR-based Filtering
    * Action: Compare text content of original vs. SR patch.
    * Criteria: Keep only pairs with minimal edit distance to avoid
      introducing distortions.

   |
   v

 4. Text Segmentation via SAM-TS Model
    * Note: This ensures masks are generated for the final, high-
      quality text, guaranteeing perfect alignment.

   |
   v

 5. Reliability Control: Further OCR-based filtering to validate
    the quality and accuracy of the generated segmentation masks.

   |
   v

The synthesis process combines three distinct data sources:

    (A) High-Quality Text Patches with Aligned Segmentation Masks
        (The output from Step 5 above)

    (B) Existing Text Segmentation Datasets
        (To enrich variety)

    (C) Clean, Text-Free Background Images
        (Derived from the LSDIR Dataset after OCR-based filtering to
        remove any images containing pre-existing text)

    |
    | All three sources (A, B, C) are fed into the next step
    |
    v

 6. Synthesis Step
    * Method: Random Pasting or Image Blending

   |
   v
[FINAL OUTPUT]
HR Images with Multiple Text Regions and Paired, High-Fidelity Segmentation Labels.

We will ensure that a clear, professionally drawn flowchart is included in the main body of the paper in the next revision.

Q3: Pasting text onto other images may not accurately reflect real-world scenarios.

A3: We appreciate the valuable question. The primary motivation for proposing the FTSR dataset was the profound scarcity of high-quality, large-scale datasets with bilingual text scenes suitable for image super-resolution. As far as we know, the available resources are either text recognition datasets with inconsistent image quality (e.g., CTR [1]) or text segmentation datasets with limited scene diversity (e.g., BTS [2]). To facilitate dataset scaling, we adopted this simple yet effective heuristic pasting strategy, which allows us to generate data with greater diversity in scenes and text combinations than is currently available.

It is important to note that, in the short term, the goal of synthetic data is not to perfectly replicate but to approximate real-world data distributions. A model trained on this data can learn the core assumption of our task—the ternary relationship between a low-quality image with text $**x**_L$, its high-quality counterpart $**x**_H$, and the corresponding ground-truth segmentation mask $**s**$, where the mapping from $**x**_L$ to $**x**_H$ must restore sufficient text-structural fidelity to accurately predict $**s**$. From this perspective, whether the transition between the pasted text patch and the background is perfectly seamless does not fundamentally alter this core relationship. The primary purpose of this large-scale synthetic data is to reduce the dependency on vast amounts of real-world data and therefore lower data acquisition costs. However, for practical deployment, a final fine-tuning step on a smaller set of real-world data is always anticipated.

In the long term, the ideal synthetic dataset should indeed simulate the real world as closely as possible to further narrow the gap and reduce the need for real-world fine-tuning. Our current strategy can certainly be improved. We have identified two promising directions for future work:

1. Image Harmonization: Employing techniques like Poisson blending to create a smoother, more natural transition between the text patches and the background images.
2. Text Image Rendering based on Generative Models: Using text-to-image models with strong text rendering capabilities (e.g., JoyType [3], AnyText [4], Seedream [5]) to generate entire scenes.

We have conducted preliminary experiments based on both of these approaches:

By applying Poisson blending to a subset of our training data, we observed a marginal improvement in perceptual quality, although the impact on quantitative metrics like PSNR and OCR-A was minimal. This suggests that while harmonization can enhance visual appeal, our core pipeline already provides the essential structural information needed for effective training.

Regarding generative text image rendering methods, our specific experimental findings are as follows:

• JoyType relies on manually rendered text outlines, which makes it challenging to scale for large-scale dataset generation.
• AnyText requires manual specification of text placement and frequently produces instances with missing or distorted strokes, or text that does not strictly adhere to the prompt. This limits its practical applicability for high-quality ground-truth generation.
• Seedream 3.0 is more promising, as it can generate high-resolution, bilingual images that largely conform to text prompts without requiring manual text placement. However, in our experiments, we still encountered issues such as text positioning inaccuracies relative to the prompt, structural distortions in small characters, and sharpening artifacts at text edges. These issues could potentially compromise the naturalness and fidelity of the text regions in full-image super-resolution results.

In summary, current generative text image rendering models are still some distance from consistently producing high-quality, full-image text ground-truths suitable for large-scale SR training. We will add the discussion in the next version of our paper to thoroughly evaluate the applicability of different data synthesis pipelines for full-image text super-resolution data generation.

Thank you again for your constructive feedback. We hope our responses have addressed your concerns.

References:

[1] Benchmarking Chinese Text Recognition: Datasets, Baselines, and an Empirical Study.

[2] A Bi-Lingual Benchmark for Text Segmentation in the Wild.

[3] JoyType: A Robust Design for Multilingual Visual Text Creation.

[4] AnyText: Multilingual Visual Text Generation And Editing.

[5] Seedream 3.0 Technical Report.