VChangeCodec: A High-efficiency Neural Speech Codec with Built-in Voice Changer for Real-time Communication

Dear Reviewer 2ct9,

We appreciate your positive feedback and find the questions and suggestions to be extremely valuable and constructive! Let us respond to your questions point by point.

Q1: I believe the evaluation is not comprehensive and it would be nice to add more target timbre.

First, we acknowledge the reviewer's concerns, which are indeed valuable. The key difference between our approach and traditional voice conversion (VC) models lies in their target audiences and operational contexts. Our lightweight VChangeCodec is embedded within real-time communication systems, where the encoder and decoder are immutable and maintained by mobile network operators.

Our method adheres to a "one model for one timbre" approach, utilizing a lightweight model tailored for deployment in mid-range performance smartphones. Therefore, our approach is a completely new approach targeting the RTC scenario with distinctive technical requirements and design.

Secondly, we have carefully considered privacy issues for applications in RTC services. Our VC module is integrated into the speech communication system, prohibiting users from accessing or altering internal configurations. The embedded speaker representation is injected at the sending end and is also maintained by the operators, with designated timbres being pre-defined and inaccessible to ordinary users. A potential use case is expected the user can only download the related binary file after subscribing to a specific target timbre from the operator. Consequently, the user cannot change the source voice to any target timbre arbitrarily to minimize the deepfake risk to operators. In contrast, previous VC models, if positioned on the user side, would allow users to arbitrarily modify the pre-defined timbres before they pass through the sender's encoder. The converted timbre, then transmitted through the operator's codec, could raise issues of timbre infringement.

Importantly, we have provided additional clarification of "Regarding the differences with zero-shot VC solutions and the limits of target timbre" in general response. We hope that the reviewers will take time to peruse these insights. Additionally, we have added a new target timbre 2 from the open-source VCTK dataset in a demonstration page. We would greatly appreciate your time in comparing the subjective quality at the following link:

https://anonymous666-speech.github.io/Demo-VChangeCodec/

Q2: Some codec baseline systems need to be replaced, the author claims comparison with SOTA codec models in Table 1, while some baselines are proposed in 2012 or 2014, it is not convincing.

We appreciate your observation regarding the selection of baseline systems. We understand your concern that some of the baseline systems mentioned in Table 1 including Opus 2012 and EVS 2014. In this section, we supplement additional material to explain why we incorporated OPUS and EVS into the comparison. We also have appended an extra comparison with the SOTA neural codec (DESCRIPT AUDIO CODEC, DAC, Neurisp 2024).

• First, the RTC speech coding standards widely applied in telecommunication and internet society are Opus (2012) and EVS (2014) (traditional signal processing-based) with high quality of experience including high quality, low latency and low complexity. And current voice calls and video conferencing services have been widely utilizing these two standards. To substantiate that our proposed method is competitive in absolute high quality with higher compressional efficiency, we have compared it with these established standards. A gentle reminder that other neural coding methods, such as Soundstream [1] and Encodec [2], have similarly been benchmarked against these established standards in their experiments.

• Secondly, the DAC currently represents the state-of-the-art (SOTA) performance in the neural speech coding field. Our comprehensive objective evaluation in Table 1 provides a rigorous comparison. We utilized POLQA/P.863 metrics, recommended by the International Telecommunication Union (ITU) and recognized as more comprehensive than traditional PESQ. Please refer to the official statement by ITU-T (https://www.itu.int/rec/t-rec-p.862). According to the empirical result, we have achieved comparable quality while reducing model parameters by a remarkable 70x in Table 4. These objective evaluations have validated the effectiveness of our lightweight, streamable codec design for real-time communication scenarios.

Furthermore, we have incorporated the two systems including NaturalSpeech 3 (FACodec, ICML 2024) and SpeechTokenizer (ICLR 2024) as suggested in Q3 and Q4.

[1] Zeghidour N, Luebs A, Omran A, et al. Soundstream: An end-to-end neural audio codec[J]. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2021, 30: 495-507.

[2] Défossez A, Copet J, Synnaeve G, et al. High fidelity neural audio compression[J]. arXiv preprint arXiv:2210.13438, 2022.

Q3: The author should present the difference between VChangeCodec and two related works inlcuding SpeechTokenizer and NaturalSpeech 3, they also are codec models and can achieve voice conversion. A comparison in experimental evaluation is necessary.

First, we must emphasize our innovations in neural codec field and operator-oriented backgrounds. Our VChangeCodec is a lightweight, low-latency codec tailored for mobile network operators. We target deploying the codec through an operator rather than via peer-to-peer communication (as detailed in lines 96-101). Our method has strong RTC features (employing a fully streaming architecture) to fulfill the requirements in low complexity, which is not achievable with codecs like Descript Audio Codec (DAC, Neurips 2024), NaturalSpeech 3 (FACodec, ICML 2024), and SpeechTokenizer (ICLR 2024) in their current designs. For example, the DAC codec has 75 million parameters, and the FACodec has 1 billion parameters.

Moreover, these highly complex codecs lack a streaming structure, rendering them unsuitable for real-time communication scenarios. Their computational make them impractical for deployment on resource-constrained devices like smartphones. Our design specifically addresses these constraints by providing a lightweight, streaming-compatible solution optimized for mobile platforms.

To enhance the credibility of our results, we conducted objective evaluations by comparing our method with the NaturalSpeech 3 (FACodec) system for two specific target timbres. This comparative analysis provides a comprehensive assessment of VC performance across different systems. The objective evaluation results are presented in the table below. Compared with the latest DDDM-VC and FACodec, our objective quality still remains competitive. Bold indicates the best result, and italics indicates the second best result.

Table Comparison with SOTA VC methods on Male timbre (Correspondence Table 2)

Table Comparison with SOTA VC methods on Female timbre (Correspondence Table 6)

Dear Reviewer 2ct9:

We would like to thank you once again for taking the time to review our manuscript! Your comments and feedback are highly appreciated.

We have provided a detailed response to every comment. In our general response, we have emphasized our innovation and privacy insights. Importantly, we have added the latest models (DDDM-VC, NaturalSpeech 3) and provided a subjective evaluation.

We would like to know whether our rebuttals addressed your previous concerns about the more target timbre and SOTA VC models. If they did, we would greatly value an increase in the score. Please, let us know if you have any follow-up concerns or comments. We would be happy to clarify those.

Dear Reviewer 2ct9,

Let us explain "why does VChangeCodec work well?". Here, we will present an intuitive explanation.

First, high-quality speech reconstruction is achieved using a pre-trained codec with less than 1 million parameters. Let $X$ denote a source segment and $T$ is a target segment. $\hat{z}_x$ denotes the token of source segment. So the transformation chain is: $X$ --> $z_x$ --> $\hat{z}_x$ --> $\hat{X}$ ~= $X$. The objective is to minimize the reconstruction loss in the waveform domain:

argmin || \hat{X} - X|| \tag{1}

argmin || \hat{T} - T|| \tag{2}

In the token domain (dimension $N =84$), we aim to minimize the difference between the $\hat{z}_x$ and $\hat{z}_t$, to achieve the best reconstruction of $T$ after the decoding process.

The token contains all content and speaker information $u_s$ for perfect reconstruction. To realize perfect voice conversion, we need to keep content information and modify the target speaker information $u_t$. For specific target speakers, we refer to the SOTA voice conversion model to generate nearly parallel data from source utterances, which keeps the content information.

We design a causal projection network to learn this timbre adaption $f$. We introduce a token commitment loss to ensure token-level minimization.

argmin || f(\hat{z}_x) - \hat{z}_t|| \tag{3}

This leads to:

argmin || \hat{z}_{t} - \hat{T} || \tag{4}

Combined with equation (2), we can obtain the high-quality target speech reconstruction. Moreover, we utilize the acoustic features extracted by openSMILE, which can represent speaker identity (timbre information) and capture the subtle emotional variations in speech.

In an ideal case, the output token $\hat{z}_t$ does not contain any information about the source speaker $u_s$. This elegant design enables VChangeCodec to achieve high-quality VC with less than 1 million parameters, while the lowest latency and almost causal VC (AC-VC) in our knowledge requires at least 2 million parameters.

Ronssin D, Cernak M. AC-VC: non-parallel low latency phonetic posteriorgrams based voice conversion[C]//2021 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU). IEEE, 2021: 710-716.

Dear Reviewer 2ct9,

We greatly appreciate your follow-up concerns and timely feedback. To comprehensively evaluate our model's competitiveness, we conducted fair experiments by retraining selected baseline VC systems with our target timbre dataset (Male timbre, correspondence Table 2). While Diff-VC and FACodec were excluded due to unavailable f0/spectrogram extraction scripts and training procedures respectively, we focused on three advanced systems: VQMIVC, QuickVC, and the recent DDDM-VC.

Retraining details are as follows: VQMIVC: Trained from scratch for 500 epochs using extracted f0 and spectral features from target male timbre data. QuickVC: Fine-tuned for 3200k steps on their provided base model at 1200k steps (due to significant training time requirements). DDDM-VC: Trained from scratch for 200k steps with offline-extracted f0.

The objective evaluation results are presented in the table below. Bold indicates the best result, and italics indicate the second-best result. VQMIVC showed decreased performance, likely due to the need for retraining a dataset-specific vocoder, which requires more complexity. Considering the limited time, we did not retrain the vocoder. QuickVC demonstrated significant improvement in timbre similarity, though still not reaching our performance. DDDM-VC showed improvements across all objective metrics. Nevertheless, VChangeCodec maintains competitive timbre adaptation capabilities and voice quality. Crucially, our system achieves these results with full streaming capability and a lightweight architecture of less than 1 million parameters. We included these experiments of the retrained target timbre in Tables 10 (Appendix A.9).

Moreover, we intentionally designed our system for any-to-one voice conversion, as this matches our target application scenario where the target timbre is exclusively maintained by mobile network operators. By deliberately not implementing any-to-any conversion capabilities, we mitigate potential privacy risks, as regular users cannot modify or access unauthorized target timbres.

We have substantially enhanced our paper with comprehensive evaluations and fair experimental settings. We would like to emphasize that the primary contribution of our work lies in the novel speech codec architecture with VC features. The evaluation against zero-shot voice conversion models has already substantially demonstrated the competitive voice quality capabilities of our VChangeCodec. We believe that this comprehensive experiment will alleviate your concerns.

Table Comparison with the retrained/finetuned SOTA VC methods on Male timbre (correspondence Table 2)

We welcome any further questions you may have and would be more than happy to provide prompt responses. We sincerely hope you will consider re-evaluating the merits of our work.

Dear Reviewer 2ct9:

We value active discussions with you. I hope you will take the time to read our new experiments using retrained versions of the other state-of-the-art methods above. This is a fair experiment to support our previous conclusions. We sincerely hope you will consider re-evaluating the merits of our work.

In general, our approach is a completely new approach targeting the real-time communication (RTC) scenario with distinctive technical requirements and design, in which the technical problems and requirements are quite different from generalized voice conversion methods.

First, the key difference between our approach and traditional voice conversion (VC) models lies in their target audiences and operational contexts. Our lightweight neural speech codec is embedded within RTC systems, where the encoder and decoder are immutable and maintained by mobile network operators. Consequently, it is not reasonable to compare with VC models on the aspect of the performance in VC only but to compare the performance and the merit of our method, comprehensively. In our opinion, current VC models are hard to fulfill the RTC requirements. The transmission process of VC in RTC is shown in Figure 1 (a), placing a lightweight VC model either as a pre-processor or post-processor of the codec (e.g., VC-->Codec or Codec-->VC) would inevitably introduce double latency for both VC processing and compression. For example, the lightest AC-VC model [1] exhibits an algorithmic delay of 57.5 ms and at least two million parameters when running on a CPU infrastructure. Considering the additional 10 ms delay of the LPCNet pitch predictor in this system, and the 40 ms delay of the speech codec, the delay will reach a cumulative latency of over 107.5 ms. This significantly exceeds the acceptable latency threshold for RTC requirements.

Secondly, our approach fundamentally differs from this lightweight VC model in methodology and efficiency. Instead of treating VC as a separate process, we integrate timbre adaptation directly within the codec transmission pipeline in Figure 1 (b). Our method maintains just a 40 ms algorithmic delay without introducing any additional latency and less than one million parameters. This novel framework represents a paradigm shift from traditional cascaded VC-codec systems to an integrated, efficient solution that achieves timbre adaptation and compression in a single unified pipeline.

Thirdly, we have carefully considered privacy issues for applications in RTC services. Our VC module is integrated into the speech communication system, prohibiting users from accessing or altering internal configurations. The embedded speaker representation is injected at the sending end and maintained by the operators, with designated timbres being pre-defined and inaccessible to ordinary users. A potential use case is expected the user can only download the related binary file after subscribing to a specific target timbre from the operator. Consequently, the user cannot arbitrarily change the source voice to any target timbre to minimize the deepfake risk to operators.

We welcome any further questions you may have and would be happy to provide prompt responses.

Dear Reviewer 2ct9:

Thank you again for your valuable suggestions about the latest baselines and guidance throughout this review process. I hope you will take the time to read our new experiments using retrained SOTA VC methods (Please refer to the previous response box). This is a fair experiment that fully addresses your concerns and further supports our previous conclusions.

Importantly, we have carefully integrated all experiments into the main body of the paper. We sincerely invite you to take some time to read our revised PDF. You should be able to see it by clicking the top-right PDF button. We used the blue font to highlight the new content. We revised the manuscript according to your suggestions as follows:

• We have added subjective evaluations and a discussion of speech codec (in lines 385-390).

• We have added subjective experiments in Table 3 (in lines 432-447) and Table 8 (in lines 912-929).

• Additionally, we have added the experiment on retrained VC models in Table 4 and the corresponding analysis including more details about model parameters of VC baseline systems (in lines 449-465).

We sincerely hope you will consider re-evaluating the merits of our work. We hope we addressed all your concerns, and in this case would be happy if you could consider increasing the score, and if not we are more than willing and happy to engage in a discussion with you to answer further questions.

Dear Reviewer 2ct9,

We sincerely thank you once again for your effort in reviewing our manuscripts. We kindly hope you could find some time to review our latest responses.

We have worked to address your concern through additional experiments and clarifications. We outline them below:

• [Fair comparison with our target timbre dataset] As you requested, we conducted additional experiments by comparing our VChangeCodec with SOTA VC models. Using the same training dataset of the selected target timbre, we ensured fairness and consistency throughout the evaluations. Such experiments can be viewed as an equivalent comparison to any-to-one models.

• [SOTA performance in all evaluations] We have demonstrated the superior performance of our VChangeCodec through extensive experiments in Table 4.

• [Key difference between our approach and traditional VC models] Our lightweight neural speech codec is embedded within RTC systems, where the encoder and decoder are immutable and maintained by mobile network operators. Consequently, it is not reasonable to compare with VC models on the aspect of the performance in VC only but to compare the performance and the merit of our method, comprehensively.

• [The updated PDF manuscript] We have carefully integrated all experiments into the main body of the paper.

We hope these substantial improvements will be considered in the final evaluation of our paper. We remain open to further discussion to enhance our work and welcome any additional questions you may have.

Thank you once again for your constructive feedback.

Sincerely,

Authors.

Dear Reviewer Y2Se,

We sincerely appreciate your positive feedback and are pleased to see your deep understanding of our paper's content and objectives. We will respond to your questions point by point.

Q1: The main weakness is the motivation which I fail to understand at this point. If a light-weight Voice conversion model can be used a post-processor or a pre-processing module after/before codec, then what additional advantage does this framework brings.

• First, placing a lightweight VC model either as a pre-processor or post-processor of the codec (e.g., VC-->Codec or Codec-->VC) would inevitably introduce double latency for both VC processing and compression. For example, the lightest AC-VC model to our knowledge [1], when run on a CPU, exhibits an algorithmic delay of 57.5 ms. Considering the additional 10 ms delay of the LPCNet pitch predictor in this system, and the 40 ms delay of the speech codec, the delay will reach a cumulative latency of over 107.5 ms. This significantly exceeds the acceptable latency threshold for real-time communication requirements.

• Secondly, our approach fundamentally differs from this lightweight VC model in methodology and efficiency. Instead of treating VC as a separate process, we integrate timbre adaptation directly within the codec transmission pipeline. By incorporating a causal projection network into the existing speech codec architecture, our method maintains just a 40 ms algorithmic delay without introducing any additional latency. Furthermore, while conventional SOTA VC methods operate on waveform-to-waveform mapping, which is computationally intensive and impractical for mobile devices, our approach processes low-dimensional tokens, significantly reducing computational complexity. This novel framework represents a paradigm shift from traditional cascaded VC-codec systems to an integrated, efficient solution that achieves both timbre adaptation and compression in a single unified pipeline.

[1] Ronssin D, Cernak M. AC-VC: non-parallel low latency phonetic posteriorgrams based voice conversion[C]//2021 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU). IEEE, 2021: 710-716.

Dear Reviewer Y2Se:

We would like to thank you once again for taking the time to review our manuscript! Your comments and feedback are highly appreciated.

We have provided a detailed response to every comment. In our general response, we again emphasized the novelty of our approach. We added comprehensive subjective evaluations in Table 7 and 8 (Appendix A.7) and the reference (Appendix A.10).

We would like to know whether our rebuttals and new PDF revision addressed your previous concerns. If they did, we kindly ask you to consider updating your score to reflect the improvements made. If there are any questions, we would be happy to provide additional details.

Dear Reviewer Y2Se:

We appreciate your insightful response and constructive suggestions. Considering the page limit, we have tried our best to reasonably incorporate new content into the main body of the paper. We sincerely invite you to take the time to check the latest PDF revision. We used the blue font to highlight the new content. We believe that the new version can meet the requirements of the ICLR conference, with complete experiments that are reproducible.

• Specifically, we have added the discussion about latency and motivation in the introduction (in lines 83-87 and 92-94).
• We have added subjective experiments (including the latest baseline system) in Table 3 (in lines 432-447) and Table 8 (in lines 912-929).
• Additionally, we have provided a more detailed experimental discussion and experimental details (in lines 426-431 and 864-886).
• We have also incorporated the suggestions of the other reviewers into the paper.

All of these modifications make our article more complete and reproducible. Therefore, these improvements significantly enhanced the manuscript's academic depth and comprehensiveness of our manuscript.

Your suggestions have inspired us to further improve the quality of our article. It is because of your support that our work can progress and continuously improve. If the new PDF version resolve your concerns, we would greatly value an increase in the score. If you have any new questions, we'll be happy to continue addressing them.

Dear Reviewer Y2Se,

We sincerely thank you once again for your effort in reviewing our manuscripts. We kindly hope you could find some time to review our latest responses.

As you requested, we have carefully integrated all details on latency, motivation, and subjective evaluation results into the main body of the paper. We hope these substantial improvements will be considered in the final evaluation of our paper. We remain open to further discussion to enhance our work.

Thank you once again for your constructive feedback.

Sincerely,

Authors.

Dear Reviewer Y2Se,

As the discussion phase is approaching the end, we sincerely hope you could find some time to review our latest responses. We hope to fully address your concerns.

We understand that your time is valuable and you may be busy with other things. However, your insights would be extremely valuable for improving our work. We greatly appreciate your consideration.

Dear Reviewer UYJc,

We truly value your positive feedback and are glad to see your constructive suggestions. We will address each of your questions in turn.

Q1: Not all final parameters are explicitly described. I wonder about the R parameter.

We appreciate the reviewer's request for clarification regarding the parameter R and the mechanism for achieving the target bit rate. We will provide a detailed explanation in our revised manuscript.

Entropy is the same as the common sense in Information Theory, i.e., the average number of bits for coding each symbol in the codebook. For example, the codebook is composed of $\{-1.0, -0.5, 0, 0.5, 1.0\}$ with codebook size 5. The Shannon entropy H(X), is defined by $H(X) = -\sum p(x_i) * \log_2 p(x_i)$, where the $p(x_i)$ is the possibility of the $i^{th}$ symbol in the above codebook.

In this paper, the value of R is 2. For the 84-dimensional codec model, using a codebook size of 5, the bit rate calculation using Shannon's formula is $−84 × log_2​(1/5)×50/1000=9.75$ kbps. Since we are considering a uniform distribution where entropy is maximized, the actual bit rate will be lower, at 9.5 kbps. For the 56-dimensional model, also with a codebook size of 5, according to Shannon's formula, $−56×log_2​(1/5)×50/1000=6.5$ kbps, and the actual bit rate in practice is 6 kbps.

We have supplemented the parameter of the relevant bit rates in the Appendix (A.3).

Q2: I suggest adding an explicit description of the differences in training data and sample rates when discussing the performance comparisons.

We appreciate the reviewer's suggestion regarding the configuration of baseline codecs (DAC and Encodec). We have added comprehensive details about the baseline codecs, including sampling rates and specific model versions. Due to page limitations, these detailed descriptions have been included in the appendix.

First, our VChangeCodec is a lightweight, low-latency speech codec tailored for mobile network operators. We target deploying the codec through an operator rather than via peer-to-peer communication (as detailed in lines 96-101). We focus on Real-Time Communication (RTC) applications such as online meetings and voice calls, where speech is the primary content. Furthermore, we also add some mixed speech (including speech with background noise and speech with music) for robustness. Moreover, DAC and Encodec are trained on extensive audio (speech, music) datasets, which generally leads to better generalization on unseen test data. In this context, our approach achieves comparable quality when evaluated on speech datasets, providing a fair basis for comparison within our target domain.

Secondly, for RTC service, 16kHz is the standard sampling rate, as the voice is the primary component in-service. Higher sampling rates are typically for audio (including music), but we focus on speech, operating at a sampling rate of 16kHz. For fair comparison in our experiments, systems (DAC, Encodec, Lyra2) with higher sampling rates were downsampled to 16kHz to ensure consistent evaluation conditions. Specifically, we used the official Encodec versions [1] for 12 kbps (n_q =16) and 24 kbps (n_q=32), with the model's sampling rate at 24kHz. We downsampled the original test audio from 48kHz to 24kHz for input into the encoder model and downsampled the output speech to 16kHz to compare the quality at the same sampling rate. Similarly, for DAC [2], we used the official configured at 16kHz, the default 8kbps model for inference. For Lyra2 [3], we conducted evaluations using the official 16kHz, default 6kbps and 9.2kbps model for inference.

Finally, our model is comparable to DAC (the number of parameters is reduced by 70x) in objective speech quality and completely surpasses Encodec at 24kbps.

[1] https://github.com/facebookresearch/encodec

[2] https://github.com/descriptinc/descript-audio-codec/tree/main

[3] https://github.com/google/lyra

Q3: Add the detailed description of the data sources.

• Regarding the training set in original voice mode, we use DNS challenge 2020 dataset and the LibriTTS dataset as the speech part of the training set. As introduced in the last question (Q2), extra mixed speech utterances (mixed with noise or music) are also included, in which the noise clips are from the DNS challenge, and music clips are from MIR-1k and FMA. This configuration is designed for actual RTC scenarios, including pure voice communications, or voice with background interference (e.g. office noise, background music playback, etc). We randomly selected noise crops and adjusted the mixing gain of the noise component using SNR, and the target SNR is 15 dB. $SNR = 10 log10 (S^2 / (kN)^2)$, S denotes clean signal energy and N is noisy signal energy.

• Recognizing that neural speech coding is inherently data-driven, we incorporated mixed speech segments, including those with noise or music, to enhance the robustness of our scheme.

• For all training data, we randomly sample 98% of the dataset for train, 1% for valid and 1% for test. Finally, the unseen test set is strictly an out-of-domain dataset, ensuring it was not exposed to any model during training. In line with the ITU-T P.800 recommendation and adhering to the codec evaluation standard [4], our test set is used to assess all codec models on an identical, unseen test corpus.

[4] Muller T, Ragot S, Gros L, et al. Speech quality evaluation of neural audio codecs[C]//Proc. Interspeech. 2024.

Q4: Give a complete picture of the performance of your codec compared with DAC and Encodec.

We appreciate the reviewers' inquiries and understand this question.

Let us clarify why DAC and Encodec were not included in Figure 3. Specifically, our method addresses Real-Time Communication (RTC) requirements through a fully streaming architecture with low computational complexity. Current implementations of codecs like Descript Audio Codec (DAC, Neurips 2024) and Encodec (2022) face significant limitations in meeting these RTC requirements. Specifically, DAC's architecture, with its 75 million parameters (VChangeCodec only needs 1 million parameters), does not support streaming inference, making it unsuitable for real-time applications. While Encodec does offer streaming capabilities, its processing speed (measured in Real-Time Factor, RTF) is significantly lower than Lyra v2, which processes audio approximately 10 times faster in real-time scenarios. Additionally, we evaluated the most authoritative POLQA/P.863 objective metrics, which are recommended by ITU to surpass PESQ.

To make our assessment more complete and credible, we have supplemented the subjective evaluation results of DAC@8kbps, Encodec@12kbps, and SpeechTokenizer. We selected 10 subjects to conduct DCRMOS evaluation on 4 Mandarin corpora. Each subject compared the quality of the four systems and the reference audio, scoring them on a 1-5 scale. The results are shown in the following table. It should be emphasized that we have a streaming structure. Our subjective scores indicate that the competitive quality is achieved with the lowest delay and parameter quantity.

Table Subjective evaluation on different neural speech codecs

For further clarification, you may refer to our detailed explanation of "Emphasizing our innovations in neural codec field and operator-oriented backgrounds" in the general response.

Q5: The synthetically generated parallel training data should be mentioned earlier.

Thank you for your valuable suggestion. We have added this design rationale in our revised introduction. “The target timbre customization is achieved using near-parallel training data generated through open-source voice conversion toolkits."

While our approach uses parallel training data, this design offers significant advantages through its simplicity and effectiveness. By decreasing the dependence on the speech recognition modules (as used in QuickVC [5], FreeVC [6] etc.), we substantially reduce model complexity while maintaining high performance. Moreover, our streamable architecture enables timbre adaptation for any speech input.

[5] Guo H, Liu C, Ishi C T, et al. QUICKVC: A Lightweight VITS-Based Any-to-Many Voice Conversion Model using ISTFT for Faster Conversion[C]//2023 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU). IEEE, 2023: 1-7.

[6] Li J, Tu W, Xiao L. Freevc: Towards high-quality text-free one-shot voice conversion[C]//ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2023: 1-5.

Dear Reviewer UYJc:

We have carefully responded to your queries and questions about our work. We have comprehensively enhanced the manuscript (new PDF revision) by providing detailed descriptions of all parameters, datasets and neural codec models. We have added thorough subjective evaluations, including the latest VC baselines and our retrained model results to substantiate our conclusion. Could you please let us know if our responses have addressed your concerns?

If you feel we have successfully resolved your issues, we kindly request you consider adjusting your initial score accordingly. We hope to receive your reply, as your feedback is of significant value for the improvement of our work, and we would be immensely grateful.

Please feel free to share any additional comments you may have.

Dear Reviewer UYJc:

We value active discussions with you. I hope you will take the time to read the rebuttal material, especially comprehensive subjective evaluations, the latest VC baselines and our retrained model results to substantiate our conclusion.

If they resolve your concerns, we would greatly value an increase in the score.

Dear Reviewer UYJc:

We appreciate your constructive suggestions about including new evaluations and all the discussions in the paper. Your rigorous and detailed feedback has been helpful, it has truly motivated us to keep improving our manuscript. We sincerely appreciate the time and effort you've put into reviewing our work. Considering the page limit, we have tried our best to reasonably incorporate new content into the main part of the paper. We sincerely invite you to take some time to check the latest PDF revision. We used the blue font to highlight the new content. We believe that the new version can meet the requirements of the ICLR conference.

• First, we have added the discussion about motivation and differences with traditional VC in the introduction (in lines 83-87, 92-94 and 105-107).

• To present our experimental results more comprehensively, we have integrated the original parameter table with Table 1. We also have added subjective evaluations and a discussion of speech codec (in lines 385-390).

• We have added subjective evaluations of the voice change mode in Table 3 (in lines 432-447) and Table 8 (in lines 912-929). We also have provided a more detailed experimental discussion about objective evaluations (in lines 420-431).

• Additionally, we have added the experiment of retrained VC models in Table 4 and the corresponding discussion (in lines 449-465).

• We have also incorporated the suggestions of the other reviewers into the paper.

We hope that all these revisions have improved the clarity and presentation of our work. These improvements significantly enhanced the manuscript's academic depth and comprehensiveness of our manuscript.

We deeply value your detailed guidance on our work. If the new PDF version successfully changes your impression, we sincerely appreciate your consideration for a higher score. If you have any new questions, we would like to continue addressing them.

Best regards,

Authors.

Dear Reviewer Dy4A,

Thank you so much for taking the time to review our paper and providing valuable comments and suggestions regarding the dataset description and human subjective evaluations. We will respond to your questions below.

W1: The proposed framework is constrained to a predefined set of target speakers, which limits its extensibility.

We understand the reviewer’s concern about the extensibility of voice conversion (VC). We have provided additional clarification of "Regarding the differences with zero-shot VC solutions and the limits of target timbre" in general response. We hope that the reviewers will take the time to peruse these insights.

We would like to further clarify our methodology and motivations (refer to general response) to highlight the key distinctions between our proposed approach and these zero-shot VC methods. Our main contribution lies in proposing an innovative method with a solid embodiment and pipeline of the neural speech codec with real-time VC features, rather than solely focusing on SOTA zero-shot VC methods. Our VChangeCodec is a lightweight, low-latency codec tailored for mobile network operators. We target deploying the codec through an operator (as detailed in lines 96-101). Given the practical constraints of latency and computational complexity in our target scenarios, our proposed method is expected to be integrated into mid-range performance smartphones.

We need to clarify a few points regarding the design of a predefined set of target speakers:

• First, we have carefully considered privacy issues for applications in RTC services. Our voice conversion (VC) module is integrated into the speech communication system, prohibiting users from accessing or altering internal configurations. The embedded speaker representation is injected at the sending end and maintained by the operators, with designated timbres being pre-defined and inaccessible to ordinary users. In contrast, previous VC models, if positioned on the user side, would allow users to arbitrarily modify the pre-defined timbres before they pass through the sender's encoder. The converted timbre, then transmitted through the operator's codec, could raise issues of timbre infringement.

• Secondly, the pre-trained weights of neural networks for specific target speakers are provided by the service provider (operator), and users must subscribe to the service from the operator. For example, after subscribing, a user might receive a binary file containing the weights for target speaker A, allowing them to change their voice to the timbre of speaker A but not to speaker B, etc. The number of target speakers in the service is controlled by the operator, who is also responsible for designing new target speakers to meet user requirements. Unlike SOTA zero-shot VC systems that allow users to customize VC using just a few seconds of any target speaker's sample, our approach with predefined speaker sets fundamentally minimizes privacy risks by avoiding the transmission and storage of arbitrary speaker characteristics.

Q1: The computational cost, storage space, and availability of experiments using pre-trained speaker embeddings.

We appreciate the reviewer‘s deep understanding of the target speaker's features. Your understanding is correct regarding the vector of speech embeddings. However, the speaker extraction networks may be retrained on new corpora [1] or an adaptive network (average pooling) [2] is introduced, which will lead to additional training overhead. The openSMILE toolkit acquires a high performance against a large baseline feature set from the INTERSPEECH challenge [3]. The speaker embedding dimension is lower than the general extraction network 256, 512, etc. We have refined the description to enhance its rigor and clarity in our paper.

Our paper primarily focuses on the codec architecture and its optimization. We found that OpenSmile's feature extraction adequately serves our current objectives and effectively contributes to timbre similarity while maintaining system simplicity. We acknowledge the valuable suggestion to explore alternative speaker embedding methods, we plan to conduct comprehensive comparisons with other speaker extraction methods (such as Wespeaker [4]) in future research.

[1] Qian K, Zhang Y, Chang S, et al. Autovc: Zero-shot voice style transfer with only autoencoder loss[C]//International Conference on Machine Learning. PMLR, 2019: 5210-5219.

[2] Yang Y, Kartynnik Y, Li Y, et al. StreamVC: Real-Time Low-Latency Voice Conversion[C]//ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2024: 11016-11020.

[3] Eyben F, Scherer K R, Schuller B W, et al. The Geneva minimalistic acoustic parameter set (GeMAPS) for voice research and affective computing[J]. IEEE transactions on affective computing, 2015, 7(2): 190-202.

[4] https://github.com/wenet-e2e/wespeaker

Dear Reviewer Dy4A:

We want to thank you once again for taking the time to review our manuscript!

We value active discussions with you. I hope you will take the time to read the rebuttal material, especially the clarification of the dataset, the comprehensive human subjective evaluation and the new demo link. FYI, we have prepared general response to clarify our motivation and the limits of target timbre.

If they resolve your concerns, we would greatly value increaing the score.

Dear Reviewer Dy4A:

We hope we've successfully addressed all your concerns, and in this case, we would be happy if the reviewer could kindly consider increasing the score. If not, we're pleased to continue our dialogue and provide any additional clarifications you feel would be helpful.

Dear Reviewer Dy4A:

Thank you again for your valuable suggestions and guidance throughout this review process. Given the page constraints, we have carefully integrated all essential explanations into the main body of the paper. We sincerely invite you to take another look at our revised PDF. You should be able to see it by clicking the top-right PDF button. We used the blue font to highlight the new content. Thanks to your helpful feedback, our manuscript has been substantially enhanced. Specifically, we have made the following key improvements:

• We have added explanations about the difference from generalized VC in the introduction (in lines 83-87 and 92-94) and have included a demonstration link for better accessibility (in lines 112-113).

• We have supplemented our target scenarios and privacy issues of applications in RTC services (in lines 772-796).

• We have added the comprehensive dataset descriptions and experimental setup details (in lines 301-304 and 864-876).

• We have added subjective experiments in Table 3 (in lines 432-447) and Table 8 (in lines 912-929). We also have provided a more detailed discussion of objective evaluation results (in lines 420-431).

• Additionally, we have added the experiment on retrained VC models (any-to-one) in Table 4 and the corresponding analysis (in lines 449-465).

• We have also integrated valuable suggestions from other reviewers.

These changes not only make our work more thorough and convincing but also substantially strengthen the academic rigor of our manuscript. We believe these improvements have effectively enhanced the overall quality of our paper.

We hope we have resolved all the concerns and that the revised manuscript will better suit the ICLR conference. If so, we would be very happy if you could kindly consider increasing the score. If not, we are very willing to engage in a discussion with you. We will do our best to answer any other questions and give more explanations.

Best regards,

Authors.

Regarding the differences with zero-shot VC solutions and the limits of target timbre (for 2ct9, Dy4A).

• We have carefully considered privacy issues for applications in RTC services. Unlike zero-shot VC, which allows users to select any target speaker's timbre in online conversations, our method restricts this flexibility. It's important to note that our voice conversion (VC) module is integrated into the speech communication system, prohibiting users from accessing or altering internal configurations. Consequently, users cannot freely change voices within an RTC system. In practical applications, the service operator should maintain model parameters on the backend, and users (on the sending side) must download a specific model corresponding to a certain timbre after subscribing.

• Zero-shot VC operates on a "one model for all timbres" principle and is not optimized for low latency or low complexity on mobile devices. In contrast, our method adheres to a "one model for one timbre" approach, utilizing a lightweight model tailored for deployment in mid-range performance smartphones. Therefore, our approach is a completely new approach targeting the RTC scenario with distinctive technical requirements and design.

• Conclusively, the proposal in this paper aims to enhance the experience in real-time communication, including conversational chatting. Therefore, the extensibility of the target speaker should be properly considered due to privacy concerns. This consideration is a cornerstone of our contribution, and to the best of our knowledge, this represents the first disclosure of a paper addressing this specific aspect.

Regarding the dataset description in detail (for Dy4A, UYJc).

• We provide the following clarifications on the evaluation corpus. For the objective evaluation of the original voice mode, we utilized both Mandarin and English utterances. For the subjective evaluation (as detailed in lines 347-351), we focused on Mandarin utterances, given the subjects’ native language. We have provided both English and Mandarin utterances of the original mode in our anonymous repository.

• In the voice change mode evaluation, we assessed both Mandarin and English utterances. Notably, we chose to evaluate English utterances using Word Error Rate (WER) and Character Error Rate (CER) exclusively, as the Whisper model is less applicable to Mandarin, thus ensuring a more equitable comparison. For DNSMOS, MCD, and Resemblyzer assessments, we included both Mandarin and English utterances. We have bolstered our evaluation with the addition of N-MOS and S-MOS to substantiate our results.

• Regarding the training set in original voice mode, we used all DNS challenge 2020 dataset. Recognizing that neural speech coding is inherently data-driven, we incorporated mixed speech segments, including those with noise or music, to enhance the robustness of our scheme. Our test set is strictly an out-of-domain dataset, ensuring it was not exposed to any model during training. In line with the ITU-T P.800 recommendation and adhering to the codec evaluation standard [1], our test set is used to assess all codec models on an identical, unseen test corpus.

[1] Muller T, Ragot S, Gros L, et al. Speech quality evaluation of neural audio codecs[C]//Proc. Interspeech. 2024

Please let us know if you have any further questions. Thank you!

Demo link

Dear ACs,

Following the rebuttal guideline email, we provided an anonymous link to our demo sample to show the results in our paper:
https://anonymous666-speech.github.io/Demo-VChangeCodec/.

We will upload a new PDF file (revision) according to the reviewers’ valuable suggestions. We appreciate your ongoing interest in our research. In the following, under each reviewer's comment, we address the concerns of the reviewers point by point.

Best regards,

The 10137 authors.

Regarding to provide the complete subjective evaluations (for all reviewers).

Regarding the subjective evaluation of Codec, we have supplemented the subjective evaluation results of DAC@8kbps, Encodec@12kbps, and SpeechTokenizer. We invited ten subjects to conduct a DCRMOS evaluation on four Mandarin corpora. Each subject compared the quality of speech compressed by four systems with the reference speech, scoring them on a 1-5 scale.

The results are shown in the following table. The test result indicated clearly the competitive quality is achieved in the method we proposed even it has a streaming structure with the lowest delay and parameter quantity.

Table Subjective evaluation on different neural speech codecs

To more accurately represent voice quality and make our results more convincing, we have supplemented the subjective evaluation of our voice change mode experiments by including the latest VC benchmarks, DDDM-VC (AAAI 2024) and FACodec (ICML 2024). The evaluation involved ten subjects who assessed speech naturalness and speaker similarity on a 5-point mean opinion score (MOS) scale, ranging from 1 (bad) to 5 (excellent).

We evaluated six VC systems, focusing on two target timbres (one male and one female). The test set includes two male and three female speakers. This resulted in five conversion pairs for each specific target timbre, leading to a total of 30 converted utterances from the six VC systems evaluated by each subject. Subjects scored naturalness (NMOS) and similarity (SMOS) for 30 converted utterances for the specific target timbre. The subjective results are presented in the table below.

Table N-MOS and S-MOS on Male timbre (Correspondence Table 2)

Table N-MOS and S-MOS on Female timbre (Correspondence Table 6)

Based on our findings, we can confidently state that our model attains the highest scores in subjective evaluation for S-MOS and the near-optimal performance in N-MOS. Our method is particularly tailored to specific target timbres, offering less versatility in timbre conversion but superior quality in the conversion of selected timbres. This approach illustrates that the "one timbre, one model" strategy is not only feasible but also robust.

Additionally, we would like to highlight that the DNSMOS (an objective evaluation) has gained broad acceptance in the field, as evidenced by its use in Soundstorm (arXiv 2024) [3] and StreamVC (ICASSP 2024) [4]. Similarly, Resemblyzer is utilized by both QuickVC and StreamVC. Furthermore, the Whisper recognition model stands as the official method for assessing intelligibility in the Codec-SUPERB competition [5] and is also adopted by SpeechTokenizer (ICLR 2024).

[3] Borsos Z, Sharifi M, Vincent D, et al. Soundstorm: Efficient parallel audio generation[J]. arXiv preprint arXiv:2305.09636, 2023.

[4] Yang Y, Kartynnik Y, Li Y, et al. StreamVC: Real-Time Low-Latency Voice Conversion[C]//ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2024: 11016-11020.

[5] https://github.com/voidful/Codec-SUPERB

Regarding the new demo link for our VChangeCodec (for all reviewers):

To facilitate a more direct assessment of audio quality, we have updated an anonymous GitHub repository with a demo. Notably, we have incorporated two new voice conversion (VC) systems, DDDM-VC and NaturalSpeech 3 (FACodec) from 2024, along with English and Mandarin corpora for our VChangeCodec. We invite the reviewers to listen to the samples available at the following link:

https://anonymous666-speech.github.io/Demo-VChangeCodec/

Dear Reviewers,

We have thoroughly incorporated all valuable suggestions into the main body of the paper, including our motivation, comprehensive experimental results and detailed discussions. We sincerely invite you to take some time to read our revised PDF. You should be able to see it by clicking the top-right PDF button. We used the blue font to highlight the new content.

We would greatly appreciate it if you could review these updates. Your feedback would be invaluable for further improving our work.

Best regards,

Authors