Unlocking Speech Instruction Data Potential with Query Rewriting

Q5: Lack of Baseline Comparison: Text normalization in TTS has a rich history...

Q6: Dependency on Specific TTS Model: The study employs only Parler TTS for text synthesis. It’s uncertain how the framework would perform if a phoneme-based TTS model were used, particularly regarding out-of-vocabulary issues.

We added a text normalization method as a baseline for comparison, which achieved 98% accuracy in the Google Text Normalization Challenge.

To demonstrate the generality of our method and control the influence of speaker style descriptions on the synthesis results, the TTS models in the following experiments use a single speaker style setting during synthesis.

The experimental results are shown below, where we use SIM as a metric to evaluate the results. The experimental results demonstrate that our method consistently achieves favorable performance across different TTS models.

Which TN mean the text normalization, Phi3 mean only use Phi-3-small-8k-instruct to rewrite query, Qwen2 mean only use Qwen2-7B-Instruct to rewrite query, Llama3 mean only use Llama-3-8B-Instructt to rewrite query, and Ours w/o KF refers to using only the training-free rewriting framework illustrated in Figure 1.

For the out-of-vocabulary issues, through the query rewriting method, we can address out-of-vocabulary terms such as abbreviations, dates, and mathematical formulas.

We are deeply grateful for your recognition of our work's innovation and thoroughness, as well as their constructive feedback. We have addressed each suggestion on the manuscript's weaknesses and made the necessary revisions.

Q1: Limited Scope:...

Q2: Limited Technical Contribution:...

The query rewriting framework proposed in this paper differs from text normalization. Text normalization does not alter the word order or expression of the query itself, whereas query rewriting can rephrase sentences in new ways. This capability allows query rewriting to overcome the limitations of text normalization methods when dealing with complex texts or texts requiring additional knowledge.

The primary contribution of this paper in terms of training methodology lies in the discussion of training objectives. Previous works have used the continuation results generated by LLMs from the textual information of automatic speech recognition data as training objectives. In contrast, this paper highlights the significant differences in results when using high-quality answers versus LLM-generated continuations as training objectives, demonstrating the limitations of existing continuation-based methods.

Q3: Insufficient Details on Model Training and Evaluation: The results section ("Use synthetic data to fine-tune LSLMs") lacks clarity regarding fine-tuning methods...

We will supplement the following details in the experimental setup section:

For the training in LSLM Finetune, we set r=8 and a=32. The peak learning rate was set to 3e-5, and the cosine scheduler was used for learning rate adjustment.

The instruction templates we used during the training phase are as follows. We will include this information in the appendix of the revised manuscript.

SYSTEM

This is a chat between a user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user’s questions based on the context. The assistant should also indicate when the answer cannot be found in the context.

{doc}

User

Answer the following question with a short span. The answer needs to be just in a few words.\n

Q4: ASR Model Dependency: The method’s efficacy in cases where ASR systems fail to recognize rare words is uncertain, raising questions about the proposed solution's handling of out-of-vocabulary cases

For the out-of-vocabulary issues, through the query rewriting method, we can address out-of-vocabulary terms such as abbreviations, dates, and mathematical formulas.

We greatly appreciate the thoughtful critique and suggestions from Reviewer PEw1. Below is a summary of our revisions and clarifications based on the provided feedback.

1.Technical Contributions. This paper contributes to both data synthesis methods and training approaches. We propose a novel synthesis method for speech command datasets, which, unlike text normalization, enhances the quality of synthesized speech commands by rewriting queries while maintaining semantic consistency. In terms of training methods, our comparison of different training strategies on downstream tasks highlights that using high-quality human-annotated answers as training targets is crucial for improving the performance of LSLM. Methods based on LLM continuation limit the model’s ability to follow speech commands effectively.

2.Baseline Comparison. The work most closely related to ours is text normalization, a common industrial approach used to optimize the input for TTS models, which rewriting queries to fit the input requirements of the TTS model without altering their expression. We added a text normalization method as a baseline for comparison, which achieved 98% accuracy in the Google Text Normalization Challenge. We present the experimental results in Tables 2, 3, 14, and 15 of the revised manuscript.

3.Scalability. We considered both multi-speaker and single-speaker settings, adding two TTS models: MeloTTS and MMS-TTS-ENG. The experimental results under the multi-speaker setting are presented in Tables 2 and 3, while the results for the single-speaker setting are shown in Tables 14 and 15. The results validate the excellent scalability of our method and demonstrate its unique advantages, bridging the gap in synthesis accuracy between vocoder-based TTS models and autoregressive TTS models.

4.Experimental Details. In the revised manuscript, we provide detailed experimental setups in Section 5.1, including baselines, datasets, training methods, backbone models, and training details. Additionally, we present the instruction templates used during the fine-tuning phase of LSLM in Table 11.

We hope these revisions and clarifications address your concerns and look forward to any additional feedback or questions.

We are deeply grateful for your recognition of our work's innovation and thoroughness, as well as their constructive feedback. We have addressed each suggestion on the manuscript's weaknesses and made the necessary revisions.

Q1: The paper begins by highlighting the gap between LLM-generated responses and human responses...

The response gap between the LLM and humans mentioned in this paper primarily refers to the differences between text responses generated by the LLM and those annotated by humans. Our experiments in Table 4 demonstrate that using LLM-generated responses as training targets is insufficient compared to using high-quality, human-annotated responses.

Q2: The quality of synthetic data is primarily assessed using automated metrics...

Thank you for your suggestion. To more thoroughly evaluate the semantic consistency between our rewritten content and the original, we employed a voting mechanism using multiple LLMs to simulate human annotation. This approach verifies the consistency of linguistic information between the synthesized speech generated from the rewritten queries and the original text.

Specifically, we used Qwen2-7B-Instruct, Llama-3-8B-Instruct, and Phi-3-small-8k-instruct to validate consistency, employing the following prompts to guide the models in providing judgment results. We conducted tests on the Drop, Quoref, ropes, and TAT-QA datasets, utilizing Parler-TTS-Large-v1 as TTS Model.

# Task #

Please evaluate whether the following two sentences convey identical meanings in content. Use yes or no to give your verdict.

# Sentence 1 #

{original_text}

# Sentence 2 #

{rewrite_text}

original_text refers to the original text, while rewrite_text refers to the linguistic information of the speech synthesized via TTS after rewriting, which is obtained through an ASR model.

The following are the experimental results. We use the pass rate as the metric, where a sentence is considered to pass the validation if more than two models deem it consistent.

Q3: It’s unclear whether all embedding methods in Table 5 are used, or if only a subset based on Table 6 is applied.

In Table 6, we follow the method illustrated in Figure 1, using the mean of all embedding models as the similarity evaluation metric. We report the results of using different combinations of embedding models in Appendix C to verify that our method does not depend on specific embedding models.

Q4: The reference to "71% to 93%" lacks context. Clarifying what this range represents would improve comprehension.

The usability calculation results are derived from the mean PASS scores in the experiments, which are distributed across three tables in Tabel 5. To clarify the source of these results, we have consolidated this metric into the table below. We will include the following table in the appendix of the revised PDF.

We are deeply grateful for your recognition of our work's innovation and thoroughness, as well as their constructive feedback. We have addressed each suggestion on the manuscript's weaknesses and made the necessary revisions.

Q1: One of the main issues with this paper is ...

We report additional test results. We use different threshold settings to filter usable training samples and synthesize the test set with MeloTTS.

All test sets were validated for linguistic information using an ASR model, and any test samples that failed the validation were discarded to ensure the quality of the test set.

We use ROUGE-L as a metric to evaluate the inference results. The results are shown in the table below.]()

Threshold refers to the minimum quality score for valid training set samples, with the quality score calculated according to Equation (5).

Q2: Parts of this manuscript are poorly written and explained...

We use ROUGE-L to evaluate the model’s performance on NarrativeQA.

Q3: The implementation details in section 5.1 on page 7 also state that top-p for generation is 10 (I am assuming 10%), which is extremely low, this might as well be greedy sampling.

We carefully reviewed our code and found that during inference, we used the default settings provided by Qwen2-Audio-7B-Instruct rather than the default settings of the inference library. The specific settings are shown below, and we will correct these settings in the revised manuscript.

{
  "chat_format": "chatml",
  "eos_token_id": [151643,151645],
  "pad_token_id": 151643,
  "do_sample": true,
  "top_k": 20,
  "top_p": 0.5,
  "temperature": 0.7,
  "repetition_penalty": 1.1,
  "transformers_version": "4.38.1"
}

We will revise the settings in the new manuscript to:

For the generative evaluation phase, we evaluated our results on a single NVIDIA A40 GPU, setting top-p to 0.5, top-k to 20, repetition penalty to $1.1$ and temperature to 0.7.

Q4: Furthermore the use of Knowledge Fusion is not very well...

The proposed synthesis framework is a two-stage framework, with the first stage illustrated in Figure 2 and the second stage in Figure 3.

During the knowledge fusion stage, we used LoRA to train separate rewriting models for each dataset to rewrite the queries that were not successfully rewritten in the first stage. The training data consisted of samples successfully rewritten in the first stage. The criteria for selecting these successful samples are detailed in Section 4.3.

Q5: Furthermore, the abstract, introduction and conclusion all repeat this statement....

The usability calculation results are derived from the mean PASS scores in the experiments, which are distributed across three tables in Tabel 5. To clarify the source of these results, we have consolidated this metric into the table below. We will include the following table in the appendix of the revised PDF.

Speech:Is Bob more or less likely to be having his body respond to his current temperature than Gretchen?

Qwen2-Audio-7B Output:Is Bob more or less likely to be having his body respond to his current temperature than Gretchen?

Qwen-Audio-7B Output:Is Bob more or less likely to be having his body respond to his current temperature than Gretchen?

Speech:Which of the two most likely has their blood and sweat vessels dilated?

Qwen2-Audio-7B Output:Which of the two most likely has their blood and sweat vessels dilated?

Qwen-Audio-7B Output:Which of the two most likely has their blood and sweat vessels dilated?

Original: What do you know?

ASR: What do you know

For the training data to train LSLMs, we calculate the data quality using Equation (5) and set a threshold t. The quality of all data used for training the model must exceed t. 
For data derived from the same original text, only the speech instruction with the highest quality that meets the threshold requirement is included in the training.

Thank you very much for your thoughtful and detailed response. We value your insights, and in response to your points of concern, we offer the following explanations:

Regarding your experimental setup, it is still a little confusing to me the exact difference between Golden and LLM Continue. How exactly are these alignment targets?

Golden refers to high-quality human-provided answers corresponding to the speech instructions, while Continue refers to answers generated by the LLM.

Limited by the capabilities of the LLM itself, the answers it generates often differ from the true answers and can even be misleading. This increases the difficulty of model learning and may introduce incorrect training objectives.

Constrained by the inherent limitations of the LLM, the answers it generates may deviate from the true answers and even be misleading. This can increase the difficulty of model learning and potentially result in incorrect training objectives.

We present the performance of LLM-generated answers on the training set below to quantify this gap. The results are evaluated using ROUGE-L.

For the alignment method, we use the cross-entropy loss function to calculate the loss, which is computed only for the Response part in Table 11. The instruction templates used during the training process are provided in Table 11.

In the first row of Table 4, what exactly is that experiment that has no Training target or threshold or construction method?

The first row of Table 4 shows the evaluation results of the untrained backbone model, Qwen2-Audio-7B-Instruct. We have updated the table in the revised manuscript for better readability.

There also seems to not be a text normalization ablation in Table 4 either.

Table 4 presents the evaluation results on downstream tasks after training the model with the speech instruction data we constructed. TN, as a baseline data synthesis method, has its results shown in Tables 2, 3, 14, and 15. It is significantly inferior to our method in terms of data synthesis quality and underperforms in data usability (TN 82.05%, Ours 93.07%).

It is noted in the paper on lines 280-283 that

To demonstrate the effectiveness of our method on longer texts, we separately report the performance of various methods on samples with original text lengths exceeding 20 words in each dataset. We use SIM as the evaluation metric under the single-speaker setting. The experimental results show that our method maintains good performance on longer texts.

Regarding Q9, while preserving this kind of paralinguistic content can be important, it is unclear to me whether these datasets really benefit from this kind of information being preserved?

Since the embedding model considers punctuation when calculating similarity, reducing this influence helps improve the accuracy of quality assessment.

In the examples below, slight changes in similarity can be observed as punctuation changes.

In our data, due to the high average similarity between the linguistic information of synthesized speech and the original text, variations in punctuation can also affect the evaluation results.

what do you know?
SIM:1.000
what do you know
SIM:0.993
what do you know.
SIM:0.987
what do you know!
SIM:0.971

Thank you very much for your thoughtful and detailed response. We value your insights, and in response to your points of concern, we offer the following explanations:

In this paper, we assess data quality primarily based on the SIM metric, where Pass refers to the proportion of synthetic data with a SIM greater than 0.9. To verify the correlation between this metric and the performance of downstream tasks, we include the following experiments:

Specifically, we use the best ASR results corresponding to the speech data as the query, with the same instruction template, and use Llama3 to generate responses. This eliminates the impact of synthetic speech training data of varying quality on the experimental results. We mainly compare the experimental results under the following settings.

Ref: Directly using the original text as the query

Ours: The best ASR recognition results of the synthetic speech data generated by our method in a multi-speaker setting.

TN: The best ASR recognition results of the synthetic speech data generated by the text normalization method in a multi-speaker setting.

Original: The best ASR recognition results of the synthetic speech data generated by the original method in a multi-speaker setting.

The best ASR recognition results are obtained by selecting the one with the highest similarity to the original text.

We report the results on the training sets of Drop, Quoref, and Ropes.

The experimental results show that as the average SIM increases, there is a consistent improvement in performance on downstream tasks.

I thank the authors for their detailed responses.

My main point in adding that question about the paralinguistic content leads me to one of my main concerns: a majority of this paper is related to showing that this technique improves WER/SIM/PASS on the synthetically generated data, not with downstream tasks, which ultimately is the main goal. Time permitting, I would mainly like to see some kind of analysis that these metrics are strongly correlated with downstream performance, or some references to this phenomena. Perhaps some kind of correlation analysis or something to show that these examples can be a strong proxy/indicator for downstream performance.

I am raising my soundness and presentation scores accordingly, but I would need to see convincing evidence addressing my concern to raise my overall score.

Dear Reviewer 2kw3,

Thank you for your positive response and support for our work! We noticed that the rating has not been updated, so we would like to confirm this with you.

Thank you again for your time and assistance!

Q4: Parameter sensitivity analysis missing (e.g., impact of different thresholds).

We use different threshold settings to filter usable training samples and synthesize the test set with Melo to validate the correctness of our threshold selection.

All test sets were validated for linguistic information using an ASR model, and any test samples that failed the validation were discarded to ensure the quality of the test set.

We use ROUGE-L as a metric to evaluate the inference results. The results are shown in the table below.

The experimental results indicate that using 0.9 as the threshold for training set sampling demonstrates consistently good performance on downstream tasks.

Q5: Scalability and computational costs not thoroughly discussed.

Scalability

We provide a detailed report in Q2 on the performance of our method across multiple datasets and TTS models. Our method demonstrates consistently good performance on different TTS models, indicating its scalability.

computational costs

We have detailed the costs associated with the proposed speech instruction construction method and added a discussion on the costs of MeloTTS, a vocoder-based TTS model.

We primarily discussed the cost issues of several different speech synthesis methods and provided the costs of different speech synthesis methods across various datasets.

• MeloTTS/Parler+Original:Synthesizes speech using MeloTTS/Parler-TTS-Large-v1, with only the original text as input to the TTS model.
• MeloTTS/Parler+Ours w/o KF:Synthesizes speech using MeloTTS/Parler-TTS-Large-v1, following the method illustrated in Figure 1.
• Parler+Ours:Synthesizes speech using Parler-TTS-Large-v1, following the method illustrated in Figure 1, while applying the knowledge fusion method shown in Figure 2 to rewrite queries that failed in the first stage.~~

We deeply appreciate the time and effort you have dedicated to reviewing this paper. We value your insights, and in response to your points of concern, we offer the following explanations:

Q1: No direct comparison with other query rewriting methods from adjacent domains.

The work most closely related to ours is text normalization, a common industrial approach used to optimize the input for TTS models, which rewriting queries to fit the input requirements of the TTS model without altering their expression. We added a text normalization method as a baseline for comparison, which achieved 98% accuracy in the Google Text Normalization Challenge.

To demonstrate the generality of our method and control the influence of speaker style descriptions on the synthesis results, the TTS models in the following experiments use a single speaker style setting during synthesis.

For Parler-TTS-Large-v1, we use Jon's voice is monotone yet slightly fast in delivery, with a very close recording that almost has no background noise as the speaker style descriptions.

For MeloTTS, we use the officially provided ‘EM-US’ setting.

For mms-tts-eng, no additional speaker style control measures are provided by the official implementation, so we follow the default settings.

The experimental results are shown below, where we use SIM as a metric to evaluate the results.

Which TN mean the text normalization, Phi3 mean only use Phi-3-small-8k-instruct to rewrite query, Qwen2 mean only use Qwen2-7B-Instruct to rewrite query, Llama3 mean only use Llama-3-8B-Instructt to rewrite query, and Ours w/o KF refers to using only the training-free rewriting framework illustrated in Figure 1.

# Task #

Please evaluate whether the following two sentences convey identical meanings in content. Use yes or no to give your verdict.

# Sentence 1 #

{original_text}

# Sentence 2 #

{rewrite_text}

We greatly appreciate the thoughtful critique and suggestions from Reviewer TuG5. Below is a summary of our revisions and clarifications based on the provided feedback.

• Comparison of Neighboring Domain Methods. The work most closely related to ours is text normalization, a common industrial approach used to optimize the input for TTS models, which rewriting queries to fit the input requirements of the TTS model without altering their expression. We added a text normalization method as a baseline for comparison, which achieved 98% accuracy in the Google Text Normalization Challenge. We present the experimental results in Tables 2, 3, 14, and 15 of the revised manuscript.

• Scalability. We considered both multi-speaker and single-speaker settings, adding two TTS models: MeloTTS and MMS-TTS-ENG. The experimental results under the multi-speaker setting are presented in Tables 2 and 3, while the results for the single-speaker setting are shown in Tables 14 and 15. The results validate the excellent scalability of our method and demonstrate its unique advantages, bridging the gap in synthesis accuracy between vocoder-based TTS models and autoregressive TTS models.

• Computational Cost. In Table 7 of the revised manuscript, we provide a more detailed discussion on experimental costs to demonstrate the superiority of our method in terms of computational efficiency.

• Discussion on Thresholds. In Table 4, we present the impact of data quality thresholds on downstream tasks, and in Section 5.1 (Dataset), we provide a more detailed explanation of the composition of the training data.

• Human Evaluation

Generation Quality Evaluation

We selected 50 samples from Drop, Quoref, Ropes, and NarrativeQA, respectively, and had human annotators evaluate the quality of the responses. We ensured that the input speech did not contain any quality issues in the text.

During the evaluation process, we asked annotators to assess whether the model-generated results were consistent with the reference answers. The experimental results are shown in the table below. We reported the number of samples that passed the verification.

The method settings are as follows:

• Continue+Ours: The training objective uses the Continue setting, and the data synthesis method uses our proposed approach.
• Golden+Original: The training objective uses the Golden setting, and the data synthesis method uses the original setting.
• Golden+Ours: The training objective uses the Golden setting, and the data synthesis method uses our proposed approach.

These settings correspond to the training methods in rows 2–4 of Table 4.

Data Quality Evaluation

We conducted a small-scale manual annotation experiment to demonstrate the improvement in data quality achieved by our method under rigorous human-machine interaction evaluation.

Specifically, we randomly selected 100 samples from each of the four datasets: DROP, Quoref, ROPES, and NarrativeQA. We ensured that these samples had a SIM score below 50 under the original method, which indicates that there are significant differences in linguistic information between the speech data and the original text.

During the annotation process, we instructed annotators to verify whether the speech content conveyed the same meaning as the original text and provide annotation results. Any annotations completed in less time than the duration of the audio were discarded and re-annotated to ensure the validity of the annotations.

The following are our experimental results, demonstrating that our method significantly improves the linguistic consistency of synthesized speech.

We hope these revisions and clarifications address your concerns and look forward to any additional feedback or questions.