Refining Corpora from a Model Calibration Perspective for Chinese Spelling Correction

Thanks for your constructive and insightful comment.

Question 1: Generalization in the fields of pseudo-labels, self-training, co-training, etc.

Our findings indicate that models trained on OCR/ASR data perform well but tend to be overconfident. Analysis reveals that OCR/ASR methods often result in false spelling errors. Conversely, models trained on randomly replaced data exhibit better calibration. This enables the correction of biases in the OCR/ASR dataset.

Our focus is on using a well-calibrated model to correct biased datasets. Extending the method to other domains is worthy of exploration, considering that self-training and pseudo-labeling can also lead to overconfident classifications. We can add some relavant discussions on these topics.

Question 2: Introduction to baseline work

As mentioned in the review, there are several optional channel models, with many current works based on phonetics, character shapes, and other aspects.

• SpellGCN: It employs BERT to extract character representations and constructs two similarity graphs for phonetics and character shapes. These graphs capture interdependencies between characters and generate a target vector with information about interactions among similar characters.

• PHMOSpell: It extracts phonetic features, character shape features, and context-related semantic features for each character. These features are integrated using an adaptive gate learned through training. Phonetic extraction includes mel-spectrograms, while character shapes are encoded using image encoding.

• DCN: Recognizing that connected characters can form words, it employs an attention-like method to incorporate additional dependency scores for adjacent characters.

• ECOPO: It treats correct characters as positive samples and characters that the model tends to predict incorrectly (often common characters) as negative samples. It incorporates an additional contrastive loss.

• SCOPE: It introduces pinyin (phonetic transcription) as an additional supervision for prediction.

• LEAD: It also utilizes contrastive learning methods, with negative samples derived from dictionary knowledge and designed based on phonetics, vision, and meaning.

Since the paper's focus is not on model architecture design, only basic models were used. Additional descriptions of the baseline models will be provided in the revised paper.

Question 3: How spelling correction is different in Chinese from spelling correction is other languages

Firstly, there is a granularity issue. In comparison to English, Chinese characters have a significantly larger scale. English has 26 characters, while Chinese has tens of thousands of characters, each with nuanced semantics. Chinese misspellings can be likened to modifying a single character in an English word, resulting in a distinct word with a different meaning.

Secondly, Chinese input methods can be classified into two categories: shape-based encoding and pinyin encoding. Both types have a substantial user base, making phonetics and character shapes crucial factors in Chinese spelling correction.

Thanks for your constructive and insightful comment.

Question 1: Text and citation issues

Thank you for your careful reading. We have addressed the omission in Table 2. Additionally, it should be noted that the experiments involving LLM are conducted using ChatGLM-6B rather than ChatGLM2. We have included the appropriate citation.

Question 2: The prompt of the LLMs to generate the corrected results

We employed a variety of prompts, such as “Chinese spelling correction task, involving only character replacements, without additions, deletions, or extra output.” Yet none managed to consistently yield successful task execution across all samples.

Question 3: Additional information on the strategies employed to varying output lengths generated by LLMs

LLMs often make synonym substitutions, which are irrelevant for Chinese spelling correction. In our experiments, we removed samples that generated lengths that did not meet the requirements. However, even among samples with the same length, there might still be instances of synonym substitutions, leading to LLMs exhibiting higher recall and lower precision. Although reproducing these occurrences is challenging, case studies indicate that they are infrequent.

Thanks for your constructive and insightful comment.

Question 1: Other works that combine Random Replacement and OCR methods in training

To the best of our knowledge, there are no other works that combine Random Replacement and OCR/ASR methods in the field of Chinese spelling correction. In fact, there is no recognized random replacement dataset. The motivation behind our approach extends beyond simple data combination; it stems from our observation that existing OCR/ASR datasets often result in overconfidence of the model due to some false spelling errors. Our findings demonstrate that models trained on random replacement data exhibit better calibration (as illustrated in Figure 1) and suitable to train a filtering model. Thus, we construct a comprehensive random replacement dataset.

Question 2: Is there a leak of test-set when determining the threshold for filtering

In our paper, the data filtering we discussed involves utilizing models trained on random replacement data to filter the SIGHAN training set. It is important to note that the SIGHAN dataset consists of distinct training, validation, and testing sets. Aside from employing a reduced training data, our approach aligns with other reference papers that utilize the same dataset. There is no concern of data leakage in our methodology.

Question 3: Is there a reason to choose different threshold for each dataset

We acknowledge that the use of varying thresholds may raise concerns. However, in our paper, we address this issue by providing detailed explanations and conducting comparative experiments.

Firstly, our threshold search is designed to be relatively coarse, with values of 1e-1, 1e-2, 1e-3, and 1e-4. This choice is made to avoid the complexity of an exhaustive threshold search and to ensure the usability of our method.

Secondly, in section 6.5, we thoroughly analyzed the impact of different datasets on the results, as demonstrated in Figure 5. Notably, the effect of threshold variations on the SIGHAN 13 dataset differs from the other two datasets. We recognized the concern that using the same threshold in Table 2 might weaken the persuasiveness of our method. However, we are more than willing to provide individual results for the three datasets at a threshold of 1e-2 upon request.

We achieved performance surpassing the baseline across various thresholds, and at the optimal threshold, we outperformed models that employed complex designs.