Towards Irreversible Attack: Fooling Scene Text Recognition via Multi-Population Coevolution Search

W1: Combined perturbation shift problem. We thank the reviewer for the valuable feedback. The combined perturbation shift refers to the phenomenon: when we combine several successful perturbations $\delta_i^*$ to get a new perturbation $\delta^* = \sum \delta_i^*$, this new perturbation could possibly be a failed perturbation. This is the key challenge we want to solve. Due to space constraints, our explanation in Section 3.3 and Figure 2 was brief; we will provide a more intuitive illustration and detailed discussion in the camera-ready version to make the concept clearer.

W2/Q3: The high computational cost. We thank the reviewer for the valuable feedback. The computational cost mainly comes from querying the target STR model, while the DE algorithm itself only needs little computation. Also, when evaluating the coevolutionary energy function $**E**^c(p_i^j)$, we only query the model once using the combined perturbation $\Delta(\lbrace p_i^j \rbrace \cup \lbrace p_{k}^* \rbrace _{N-1})$, which has the same query cost as $**E**^p(p_i^j)$. Thus, the coevolutionary energy function does not introduce any external computational cost.

Besides, we set the total attack pixel number consistent across different multi-population settings. For example, when the population number $N=10$, the population size is $M=60$; when $N=1$, the population size is $M=600$. Therefore, no matter how many populations we used, the computational cost in one generation is consistent. In our setting, the average attack time on CRNN with CUTE80 is only 18.23 seconds, which we consider acceptable.

W3: Comparative methods. We thank the reviewer for the constructive feedback. The field of black-box attacks indeed evolves rapidly. We chose OnePixel and $\mathrm{AD}^2\mathrm{E}$ because they are well-established baselines for pixel-level black-box attacks, sharing the same evolutionary paradigm as our method. To the best of our knowledge, $\mathrm{AD}^2\mathrm{E}$ remains the state-of-the-art among pixel-level black-box attacks in the STR domain. In addition, we included recent SOTA black-box attacks such as Sparse-RS and UDUP in Table 2, although these methods are either not pixel-level or not directly tailored to STR tasks. Therefore, we believe our comparison covers both representative baselines within the same paradigm and other strong black-box attack strategies across paradigms.

Q1: Pixel value initialization. We thank the reviewer for highlighting this error. The " $\theta_{min}=(0,0,0)$ and $\theta_{min}=(0,0,0)$" should be "$\theta_{min}=(0,0,0)$ and $\theta_{max}=(255,255,255)$". We chose these two values because they are found to be most effective in the work [38].

Q2: Optimization framework. We thank the reviewer for this inspiring question. Indeed, the attack task consists of two objectives: maximizing the success rate and minimizing the image perturbation. Therefore, in future work, we can use a multi-objective optimization framework, such as NSGA-II, to find the best attack pixel values. Specifically, after we obtain the attack pixel locations via MPCS, we can use NSGA-II to search for attack pixel values that are most imperceptible. This approach complements rather than conflicts with our MPCS method, and we plan to explore it as a promising direction in future work.

Q4: Coevolution strategy. We thank the reviewer for this insightful question. As shown in Table 2, when introducing the CEF, the SR, SR*, and PR get a rise of 12.09%, 22.47%, and 19.06%, respectively. This demonstrates its effectiveness and robustness. As for the coevolution strategy mentioned by the reviewer, we believe it will enhance the attack performance with several times the amount of calculation. With CEF, we calculate the energy of an individual by generating a combined perturbation $\Delta(\lbrace p_i^j \rbrace \cup \lbrace p_{k}^* \rbrace_{N-1})$ and use it to query the target model one time. If we use the top-K individuals from other populations, there are two choices:

(1) Using all top-K individuals to generate a combined perturbation $\Delta(\lbrace p_i^j \rbrace \cup \lbrace \lbrace p_{k}^s \rbrace^K \rbrace _{N-1})$. This will increase the perturbation, making it not aligned with our final perturbation. The increased perturbation will also bring a much higher SR. So it is not a good choice.

(2) Traversing the top-K individuals to generate combined perturbations one by one. This may enhance the robustness, but since each combined perturbation needs a query time, the computational cost will increase several times.

Q5: Explanation on white-box methods. We thank the reviewer for this good question. We reproduced the white-box method [37] and found that if we set no upper bound on its perturbation size, some of the adversarial examples generated by it could be severely distorted, but the average perturbation is still small. In contrast, our method has a stable perturbation size for each adversarial example. For a fair comparison, we have to set an upper bound on the perturbation size of [37].

W1: Reduced performance on long texts. We thank the reviewer for highlighting this point. We conduct some experiments on CRNN and CUTE80 to explore the performance on long texts and short texts under the same setting. When the text length is greater than 4, the success rate (SR) is 84.96% and the perturbation rate (PR) is 63.33%. In contrast, for texts with length $\leq 4$, the SR is 89.86% and PR is 134.29%. This indicates that longer texts are indeed harder to attack, as the number of attack pixels per character decreases with text length. However, as shown in Table 4, increasing the total number of attack pixels can effectively improve performance on long texts.

W2: Attack on transformer or diffusion-based models. We thank the reviewer for this question. As shown in Table 1, we have already conducted experiments on IGTR (TPAMI 2025), which is a multi-modal STR model with a transformer architecture. As for the diffusion-based STR model, to the best of our knowledge, there is no open-sourced diffusion-based STR model to implement experiments on.

W3: Against defense mechanism We thank the reviewer for this great question. As our best-known, mainstream defense mechanisms against our attack include adversarial training, post-correction, and pre-processing. First, adversarial training adds adversarial examples to the training data to enhance model robustness. Our attack method could also benefit adversarial training by generating adversarial examples. Second, post-correction is already included in our attack setting. Our method could effectively disable the post-correction. Finally, pre-processing such as image filtering could remove most of the perturbations, but it would also distort the image quality. As for the "adaptive STR model" mentioned by the reviewer, we are not aware of a specific method with this name. We would greatly appreciate it if the reviewer could provide more details, which would help us better understand and address this point.

W4/Q1: Question about the CEF. We thank the reviewer for this insightful question. First, regarding the coevolution energy function (CEF, Eq. 7), the "global constraint" refers to the information of the best pixels from the other $N-1$ populations. Specifically, we generate a combined perturbation $\Delta(\lbrace p_i^j \rbrace \cup \lbrace p_{k}^* \rbrace_{N-1})$ and evaluate its attack performance to determine the energy of the current attack pixel $p_i^j$. Second, our method employs a DE algorithm within each population to search for attack pixel locations. As DE itself cannot guarantee finding the global optimum, our method also cannot ensure a global optimum.

W5: Application scenarios. We thank the reviewer for this valuable suggestion. A specific scenario in privacy protection is: when users upload images to social platforms, we could add tiny perturbation pixels to the images so that the private texts may not be correctly recognized and collected by malicious OCR systems. Except for privacy protection, our attack method could potentially be applied in other contexts, such as poisoning training data, misleading OCR systems, or concealing toxic or prohibited content. We will further elaborate on these potential scenarios and their implications in the camera-ready version.

Q2: Influence of the partition method. We thank the reviewer for this good question. First, when $N=20$, we use 20 attack pixels to generate adversarial examples, which leads to greater distortion than $N=10$ (default setting). Second, the partitioning strategy affects the final locations of these attack pixels, which may cause slight variations in distortion—either an increase or a decrease—though the impact is generally minor.

Q3: Obvious disturbance points. We thank the reviewer for highlighting this concern. Our work mainly focuses on searching for the attack pixel locations while the attack pixel values are set to 0 or 255. If the values of 0 or 255 can successfully attack the target model, we directly use these values; otherwise, we use DE again to search for other attack pixel values. To improve imperceptibility, we are exploring better value-search strategies. A simple approach is to start from the original pixel value $v_0$ and iteratively try $v_0 \pm 1$, $v_0 \pm 2$, and so forth until a successful perturbation is found.

Q4: The population setting. We thank the reviewer for this good question. As shown in Table 4, with the same total number of pixels ($N \times M$), a larger $N$ yields better attack performance. Thus, we recommend using a larger $N$ with a smaller $M$. These parameters can also be adjusted automatically based on the text length; for example, if we aim to perturb each character by one pixel, $N$ can be set equal to the number of characters in the image.

Q1: Average attack time with early stop. We thank the reviewer for this good question. The attack time varies across different devices. On our device with a 3090 GPU, the average attack time on CRNN with the CUTE80 dataset is 18.23 seconds w/. ES and 37.17 seconds w/o. ES. This result demonstrates the effectiveness of the ES mechanism.

Q2: Datasets selection. We thank the reviewer for pointing this out. We agree that evaluating on non-Latin scripts would further validate the generality of our method. Our current experiments rely on STR models from MMOCR and OpenOCR, which are trained on large-scale English datasets. Hence, we have only conducted evaluations on four widely used English datasets. Importantly, our method itself does not rely on script-specific assumptions, and we expect it to generalize to non-Latin scripts. To validate this, we conducted a set of comparative experiments on a Chinese text recognition benchmark (from “Benchmarking Chinese Text Recognition: Datasets, Baselines, and an Empirical Study”), comparing our method with $\mathrm{AD}^2\mathrm{E}$. Due to time constraints, we randomly select 200 images from this benchmark and apply both attacks on the IGTR model, which is trained on Chinese datasets by OpenOCR. Our method achieves a success rate (SR) of 51.85% and a perturbation rate (PR) of 34.44%, while $\mathrm{AD}^2\mathrm{E}$ achieves 38.89% SR and 21.12% PR. These results suggest that our method generalizes well to non-Latin scripts.

Q3: Metric calculation example. We thank the reviewer for the suggestion.

(1) The perturbation rate (PR) metric.

$

**PR** = \frac{1}{S}\sum_{i=1}^{S}{\frac{D(Y_i, \mathcal{F}(X_i^{adv}))}{len(Y_i)}},

$

For example, if the ground truth $Y_i$ is "ABCD", and the predicted label $\mathcal{F}(X_i^{adv})$ is "CDAB", then the edit distance between them is $D(Y_i, \mathcal{F}(X_i^{adv}))=2$. The length of the ground truth is $len(Y_i)=4$. So the perturbation rate of this sample is 50%.

(2) The success rate before/after correction (SR/SR*). For example, if the number of samples is 1000, and 800 of them are successfully attacked, then the SR is 80%. Next, we use an LLM to correct the mispredicted texts. If 200 of them cannot be corrected, then the SR* is 20%.

(3) The L2 distance metric. This metric evaluates the average L2 distance between the original image $X_i$ and the adversarial example image $X_i^{adv}$:

$

**L2** = \frac{1}{S}\sum_{i=1}^{S}||X_i - X_i^{adv}||_2.

$

We will provide such examples in the supplementary material.

Q1: Decreased SR. We thank the reviewer for highlighting this concern. The decrease in SR mainly stems from two factors: (1) the combined perturbation shift phenomenon, and (2) the early-stop mechanism. To address the first, we introduce the CEF, which effectively alleviates its impact and improves SR, as shown in Table 3. The early-stop mechanism, on the other hand, is designed to reduce the generation number (Figure 4), representing a deliberate trade-off between SR and attack time.

Besides, we would like to emphasize that our primary goal is to design irreversible attacks. In STR scenarios, dictionary-based correction is a common post-processing step, so attacks that remain effective after such correction are of greater practical significance. Thus, SR* (success rate after correction) and PR (perturbation rate) are more indicative of performance than SR (before correction). Importantly, the SR drop after correction (SR $-$ SR*) reflects attack reversibility: our method shows only a slight decrease (0.61%, 87.36% $\rightarrow$ 86.75%), whereas the second-best method suffers a drastic drop (35.51%, 95.60% $\rightarrow$ 60.09%).

Q2: Symbol definition. We thank the reviewer for pointing out this error. Indeed, the $\theta_{max}$ in equation 5 is (255, 255, 255).

Q3: Lower performance. We thank the reviewer for highlighting this concern. We believe there are three main factors contributing to the relatively lower performance on SAR+IC13. (1) The IC13 dataset consists of easy samples with normal fonts aligned horizontally, which increases the attack difficulty. (2) The SAR model is designed for irregular text recognition, which means it is naturally robust against adversarial examples. (3) Most importantly, OnePixel has no constraint on the attack pixel location, so it could allow many attack pixels to concentrate in a small region and change a single character. That explains its higher SR. However, we want to note that our perturbation rate (PR) is still the highest (55.31% vs. 41.40%), and the SR drop after correction (SR $-$ SR*) is also smaller than other methods (18.45% vs. 26.35%). As we explained in Q1, these metrics are much more important for evaluating attack irreversibility, and the results demonstrate the best irreversibility of our attack method.

Q4: Missing metric in Table 2. We thank the reviewer for highlighting this concern. We didn't calculate the SR* metric in Table 2 because it is redundant and expensive. As shown in Table 2, compared to other attack paradigms except for UDUP, which has extremely low SRs, our method achieves much higher SR (9.54% $\sim$ 38.71%) and PR (27.37% $\sim$ 67.28%). Since a higher SR (before correction) and PR (more incorrect characters) directly imply a higher SR* (after correction), we believe the SR* metric is not necessary in this table.

Due to time constraints, we compute the SR* metric for the second-best method only (Sparse-rs), as a representative comparison. On CRNN and CUTE80, Sparse-rs achieves an SR of 63.73% and a PR of 40.23%, which are 23.63% and 54.31% lower than our results, respectively. After applying text correction, its SR* drops to 22.68%, which is 64.07% lower than ours. These results indicate that a lower PR typically leads to a more significant drop in success rate after correction (i.e., SR - SR*), further supporting the advantage of our method.

Q5: The ES mechanism. We thank the reviewer for highlighting this concern. The ES mechanism provides a trade-off between attack time and SR. If the attackers want the best SR regardless of attack time, they can cancel the ES mechanism by setting the parameter "early_stop" to 0 in our provided code.