We thank the reviewer for the useful comments.

Weaknesses:

W1：For example, subjective listening tests are at a relatively small scale.
A1： In the raters' scale, we have increased the number of listeners to 20 to evaluate our model in Sections 3.2 and 5.2. In the test cases, we have also increased the test pairs to 100 from the original 10 in Sec 3.2. In Sec 5.2, based on previous methods, we believe that 60 cases from 6 Indian speakers are sufficient for our zero-shot test. Subjective tests are costly. For example, in the accent AB evaluation in only the VCTK dataset, a listener needs to listen to 200 cases. That is $C^2_5 \times 20$.
W2：Evaluating Indian to native English conversion might have better with fluent English listeners who are Indian rather than Chinese to better capture some fine details.
A2： Yes, you are right. But that is indeed costly for us, the students. In our test, the raters are college students majoring in foreign languages, who have also been informed about the characteristics of the Indian accent. For example, "Indian-accented speakers may place stress on different syllables, or their intonation may be more flat or elevated, which can make their speech sound more monotone or sing-song to native English listeners." We believe they have the ability to properly conduct accent tests. We also showcase our demo page.
W3：It could have been better if the effectiveness of the approach had been shown on a few more non-native English accents.
A3： Thank you for the advice and i am trying this. We think this framework can be easily extended to other accents with low-resource data.
W4：Paper layout can be improved.Appendix is not well-formatted and has some typos (e.g. Accemt -> Accent)
A4： Sorry about the mistake. We have already made these corrections in our updated paper.

Questions

Q1：Did the authors consider other ways of measuring the effect of the influencing factors in Table 1? For example, the edit distance between the Hubert sequences of two files.
A1：The edit distance could also be a good choice. But we think the Longest Common Subsequence Ratio (LCSR) is more intuitive. In Table 1, first, we want to explore the percentage of pure semantic tokens without speaker identity leak, and the percentage of tokens affected by the accent. When the influencing factor is speaker identity, a 75% LCSR indicates that 25% of tokens have speaker identity leakage. A lower probability of speaker leak tokens explains why the generative model works in a zero-shot evaluation. And when the accent influence is added, the LCSR degrades from 0.75 to 0.54, indicating that approximately 21% of the tokens are affected by the accent.
Q2：The subjective analysis in Table 2 has been conducted on a relatively small set of utterances and a small number of evaluators. Could you elaborate more on the reliability of the results in Table 2?
A2： We've expanded to 100 test pairs and 20 raters.
Q3：References Jiang 2023a and Jiang 2023b are the same.
A3： Sorry about this. We have corrected this in our updated paper.
Q4：VALL-E has better speaker similarity in Table 3 but not in Table 7. In Table 7 caption, please mention about the zero-shot condition (as opposed to the SPK comparison in Table 3)
A4： Our framework has better speaker similarity. In Table 3, the primary reason for lower Speaker Perplexity (SPK) but comparable values to VALLE-AC in the proposed model is that the speech generated by VALLE-AC (without conversion module) retains more accent information. This suggests that more similar speech acoustic information results in more similar speech representations. This is not fair. Therefore, we present Table 7 to compare our proposed generative-only model with VALLE-AC. To avoid misunderstanding, we directly compare our proposed generative-only module with the baselines in Table 3. The results are shown in Table 3.
Q5：Minor typing issues
A5： Sorry for such typing issues. We have corrected these.

Dear reviewer, thanks a lot for your review.
Q1：Section 3: Please consider comparing hidden units from HuBERT to PPG or other features used in literature. Otherwise the use of hidden units sounds adhoc.
A1：In our framework, we require a speech tokenizer that can extract highly semantic, highly compressed tokens. It has been demonstrated that HuBert model satisfies these requirements [1][2][3]. On the other hand, PPG is a frame-level feature that involves more speaker information leakage, making it more suitable for mapping tasks. The baseline model in AC-PPG illustrates that mapping from L2 PPGs(non-native feature) to L1 PPGs(native feature) may not be as effective.
Q2：Section 3.1: Please provide more details on the HuBERT setup. How is the model trained? How many clusters are used for hidden units identifications? These have a significant impact on the analysis results. In addition, 10 pairs don’t seem to be enough to have the conclusions on the impact of accent and speaker identity from hidden units. Please verify this on a larger dataset.
A2：Thank you for pointing this. We provide additional details in Section 3.1: "we utilize the HuBERT-Base (Hsu et al., 2021) model and a k-means algorithm with 500 clusters to extract HuBERT tokens". The pretrained model can be accessed at https://huggingface.co/facebook/hubert-base-ls960. Additionally, in Section 3.1, we've expanded to 1000 test pairs from 10 pairs. We arrived at similar conclusions, and our experimental results and analyses have been publicly commented and updated in the paper.
Q3：Section 3.2: Similarly, the results are less convincing with limited human ratings. Please consider increasing the number of pairs and the number of raters.
A3：In Sec 3.2, we've expanded to 100 test pairs and 20 raters.
Q4：Sample page: Please list an utterance that is used for acoustic prompting as well here.
A4：The utterance used for acoustic prompting is also derived from the source non-native speech. And we also add these in sample page. In our framework, we randomly select 3-second segments from the source speech to extract acoustic prompts, aiming to maintain the same speaker identity post-accent conversion. If the original speech is less than 3 seconds, we take the whole sequence of the original speech.
Q5：Section 5.2: In VCTK experiments, please use more source speakers instead of just one.
A5：We follow our baseline model AC-PPG‘s evaluation setting, which utilizes 20 test cases from a speaker in the VCTK dataset which is not parallel dataset. To explore why our model performs effectively through the LCSR metric, we require the parallel datasets like Arctic and L2-Arctic datasets.
Q6：Section 5.2: Accent: “During the AB test on accent score, the ground truth (GT) speech samples were selected from the native speaker ”bdl” in the L2-ARCTIC dataset.” Is it still the case when the non-native speaker is a female? If so, please consider using a female voice as the reference.
A6：As A4, the utterance used for acoustic prompting come from the source non-native speech to maintain the same speaker identity. The native speaker, 'bdl', serves as the ground truth speaker in our accent metric.
Q7：Section 5.4: Just curious but not required - Does the proposed correction work under one-shot or zero-shot setup? 15-mins of parallel data is already a lot in production scenarios.
A7：Our framework operates under zero-shot setup. The six test speakers are unseen by our generative module, and the conversion module has only been exposed to an Indian speaker. We do not require audios with the same content but from the same speakers with different accents. The speaker identities can be different. Furthermore, we are able to acquire 4 hours of such parallel data from the Arctic and L2-Arctic datasets.
Q8：What about accent and naturalness of the zero-shot setup?
A8：As A7, our evaluation operates under zero-shot setup.
[1] Hsu, Wei-Ning, et al. "Hubert: Self-supervised speech representation learning by masked prediction of hidden units." IEEE/ACM Transactions on Audio, Speech, and Language Processing 29 (2021): 3451-3460.
[2] Wang, Zhichao, et al. "LM-VC: Zero-shot Voice Conversion via Speech Generation based on Language Models." arXiv preprint arXiv:2306.10521 (2023).
[3]Huang, Rongjie, et al. "Make-A-Voice: Unified Voice Synthesis With Discrete Representation." arXiv preprint arXiv:2305.19269 (2023).

We thank the reviewer very much for recognizing our work and suggestions. This is the final version of our comment.

Weaknesses:

W1：From the perspective of method, introducing TF-Codec as a contribution seems independent and unrelated to the theme of paper (accent conversion or accent reduction). If the speaking module is replaced with a multi-stage generative model such as Encodec, will it affect the accentedness and speaker similarity of accent conversion?
A1： TF-Codec is the core design of our generative model which enables a single-stage causal speech generation. Compared with multi-stage speech generation based on EnCodec, it achieves lower complexity and has higher potential for low-latency applications. Besides complexity, TF-Codec generates speech with higher speaker similarity (0.502 vs 0.429 in terms of SPK）and naturalness (4.08 vs 3.84 in terms of MOS) in Table 3.

Questions

Q1：Does it remove all the duplicated semantic tokens in HuBERT in both correction module and speaking module as stated in Section 3.1?
A1：In the training and inference phase, we don't remove duplicated HuBERT semantic tokens. Somehow, it helps to retain the original speech duration. In Section 3.1, we remove duplicated semantic tokens to eliminate the effect of the duration of each word in the accent analysis on HuBERT tokens.
Q2：The formula definition in “Pretraining” in Section 4.2. What is the purpose of defining $C^{t-1}$? It seems to be not used. Does $X^{t-1}$ already include $x_{t}$or not? This part may be roughly understood, but it is not clear enough.
A2：Sorry for the mistake. We correct the formula in our paper. Please refer to Formula (1) in Section 4.2.
Q3：In Section 4.2, what is the token mask ratio and strategy in pretraining stage? These details are important to reproduce the paper. Please give more explanations.
A3：We have added an explanation in our paper in Section 4.2 Pretraining as "We have experimented with corruptions like token masking, token deletion, and token infilling and we find the masking scheme works the best. Specifically, following the masking scheme in BERT, 15% tokens have been randomly selected first. Then 80% of them are replaced with [MASK] tokens, 10% of them are filled with random tokens and the remaining 10% are left unchanged.".
Q4：According to Section 4.3, the input of speaking module is the concatenation of the prompt accented semantic tokens, the target native semantic tokens, and the prompt accented acoustic tokens during inference, right? If so, the combination of accented semantic tokens and native semantic tokens has not been seen when training the speaking module. Does this mismatch affect the performance of accent conversion?
A4：That's a good question. The setting is similar to Section 3.2. In Section 3.2, the generative model is trained to generate speech conditioned on native semantic tokens. During inference, the semantic token part is the concatenation of the native content semantic tokens and the prompt semantic tokens. The experiment compares the native prompt semantic tokens with the accent semantic tokens and the result shows similar which indicates such mismatch doesn`t affect the performance. This can also be verified in the performance of our current model on AC. Please refer to our demo: https://convert-and-speak.github.io/demo/.
Q5：In section 5, what is the difference between “proposed” and “proposed-VC”? Are they the same one? Is “the traditional AC model (Liu et al., 2020)” equal to “AC-PPG”? It should be better to present the name and settings of each model more clearly.
A5：Proposed-VC in Section 5 in previous paper is Proposed without conversion module. Sorry for the unclear expression. We correct these in Table 3 in the updated paper, in which Proposed-VC refers to Generative model(TF-Codec) and Liu. et al refers to existing best machine-learning based AC method available in the public, previously named AC-PPG.

Q6：Sec 3.2: which model is used for synthesis? this section looks like experimental results, rather than analysis.
A6：The generative model in our proposed framework is used for this analysis experiment. In this section, we evaluate if the accent feature will be extended through the in-context learning. Based on this analysis, we find the accent style can hardly be transferred through the in-context learning in the generative model, which guides us to use the original accent source as the prompt to maintain the speaker identity.
Q7：Sec 5.1: any explanation on why the same speaker "ASI" is used for both training and evaluating?
A7: We utilize the speaker "ASI" for training and use 6 Indian speakers from VCTK, ARCTIC, and L2-ARCTIC for evaluation. We have now corrected this in Sec 5.1“In the process of fine-tuning the correction model, we utilize data from speaker "bdl" as the native English speaker source ...... and data from speaker "ASI" ....... In the evaluation, we employ data from speaker p248 originating from the VCTK dataset, as well as data from five speakers, specifically ASI, KSP, RRBI, and SVBI, derived from the L2-ARCTIC dataset to assess the performance of models.”
Q8：Sec 5.1: "In the AB test for accenteness, participants first listened to native and non-native reference audio. Subsequently, they heard paired speech samples (A and B) with identical content and were asked to choose the one that sounded more native-like." -- will rater be able to infer from the voice identity instead of accent?
A8: The native and non-native reference audio initially listened to have different speaker identities compared to the test pairs. Or if you're asking whether the speaker's identity is related to the accent in the test cases? Before the test, the raters have been taught to discern differences in Indian accents. And we direct the speakers to focus solely on accent differences. These 5 speakers are all indian- accent speakers in l1-l2 arctic dataset.
Q9：Sec 5.4: This seems completely unreadable to me. I don't understand what the sentences mean, and have no idea about what "3.5, 3, 2.7" numbers refer to as they are not presented in the corresponding Table.
A9: Sorry for this mistake. We have already deleted this.
Q10：Sec 5.4.1: Does the 2% LCSR in Table 6 consistent with 54% LCSR in Table 1?
A10: Apologies for the misunderstanding due to the lack of explanation. In Table 6, "without correction model," means that in the finetuning stage, we directly use the L2-semantic token and L1-acoustic tokens with different speaker identities but with the same phoneme content to fine-tune the generative model instead of using a converted model to convert non-native semantic tokens to native semantic tokens. 2% LCSR means it does not work.

We thank the reviewer for the useful advice.
About Ethics Concerns
A：We are very grateful to the reviewer for pointing out the ethics issues in our work. We have revised the wrong description such as "distorted" pronunciation updated to "different" pronunciation in foreign accents. We use the term 'convert' instead of 'correct' in our paper, including title, abstract and main parts.
Q1：Abstract: mentioned three terms: "accent conversion", "accent removal", and "accent reduction". It would be helpful to distinguish or consolidate.
A1： Thank you for advising this. We refer to these as 'accent conversion' in our updated paper.
Q2：Abstract: "TF-Codec" used without explanation nor reference.
A2： TF-Codec is a speech neural codec which follows paper[1]. We update this in abstract "TF-Codec is a pretrained speech neural codec with group quantization which can be used as a single-stage autoregressive generation."
Q3：Sec 2: "there has not been a parallel corpus that contains pairs of audios having the same contents yet coming from the same speakers in different accents" -- better to restrict such claims as "public corpus", as you don't know proprietary ones.
A3：We update this "For accent conversion task, there has not been a public parallel corpus that contains pairs of audios having the same contents yet coming from the same speakers in different accents."
Q4：Sec 3.1: LCSR = LCS / utterance_length -- clarification is needed because the two utterance can have different lengths.
A4：We update this "The length of the parallel data in each pair is similar and the utterance length ranges from 3 seconds to 5 seconds. The smaller utterance length is used to calculate the LCSR of each pair."
Q5：Sec 3.1: Table 1 is misleading because "accent without phoneme corruption" and "accent with phoneme corruption" implies different speaker. So the trends in the table is completely expected. It cannot draw conclusion as the paper stated, that " the speaker identity causes little impact on the Hubert tokens and the content is not disturbed", "accent feature ... brings larger influence on the Hubert tokens".
A5： We update the description in my paper “As the results shown, with the accent source, HuBert tokens have changed a lot, degrade from 0.75 to 0.57 in terms of LCSR. For those cases with specific phoneme changes, more tokens have changed from their native references(LCSR: 0.54).”
[1] Jiang, Xue, et al. "Latent-Domain Predictive Neural Speech Coding." IEEE/ACM Transactions on Audio, Speech, and Language Processing (2023).