Sylber: Syllabic Embedding Representation of Speech from Raw Audio

We deeply appreciate the reviewer for taking time for reviewing and providing meticulous comments. We conducted additional experiments and analyses to address weaknesses and questions. Below, we address them one by one in detail. We also updated the manuscript accordingly.

Training efforts (W-1): The original numbers of updates in the paper, 1.15M and 500K, were mistakenly stated, and the number of updates we actually used are 115K and 50K. We sincerely apologize for the confusion. Therefore the efforts for training are mistakenly overstated. Given this correction, the second stage training is not as big a burden as it originally sounds, and this is even expedited through the proposed linear time greedy algorithm. As the reviewer commented, the second stage does not have a huge effect, but we do not particularly claim that this stage is necessary. Still, the second stage training further refined the segmentation accuracy and quality (especially precision). Most importantly, the second stage training demonstrates an important behavior that self-bootstrapping the segments doesn’t lead to degeneration, and actually can refine the syllabic structure. Therefore, we kept the second stage and used this model for the other experiments and analyses.

Model scalability (W-2): We acknowledge the known instability in training with EMA-based self-distillation, especially with a simple regression loss [1]. However, the proposed method is not as sensitive as previous approaches since we initialize the models with SDHuBERT. To further demonstrate this, we additionally trained models with a reduced training setting with two different model initializations, SDHuBERT or HuBERT. In particular, we used a less stable optimization: We decreased EMA decay from 0.9995 to 0.999, increased the learning rate from 1e-4 to 5e-4, and reduced the number of updates from 115K to 50K. We skipped the second stage training given the limited time window. Furthermore, we simulated a noisier setting by randomly adding noise to the initial segments, to make training more challenging. We randomly selected 20% of the boundaries and shifted 20-80ms, affecting 36% of the segment annotations in total. This perturbation is done once before the training.

When syllable detection and discovery scores are measured, the ablation models also show high scores near to the original Sylber (top row), and show similar prominent structures in frame-wise similarity matrices (Figure # in the updated manuscript). This higher learning rate and lower EMA decay is supposed to add instability in training, however, the models can be successfully trained without struggling. This indicates that our method is not as sensitive to hyperparameters as other self-distillation approaches. A proper model initialization is the only prerequisite we have observed since the model is not trainable from scratch. Therefore we expect instability of training would not be a major issue for scaling the model size, though it is yet to be shown whether scaling law is as important in syllable learning as other SSL models.

While scaling the model size does lead to increased costs, this is a problem inherent with the scaling of all SSL based models. Furthermore, this secondary training with Sylber can lead to more efficient and scalable models down the road. We have found that the RTF of running the segmentation using Sylber is so low that it can be done on the fly to segment audio data for further downstream segmentation distillation, so this step of the training process is not a significant bottleneck of the training. (refer to the response below (R5) about RTF or Appendix A.8.) In addition, due to the reduction of the frequency of tokens down to 4.27 tokens/s, this means that spoken language models can train on longer-form speech data more efficiently, since the sequence length required to train these models will be reduced by an order of magnitude, allowing researchers to scalably train SLMs by reducing the amount of memory and GPUs required to scale.

Difference between the discriminability index and phonetic ABX (Q-1): The purpose of the two metrics are very different. The discriminability index is targeted to measure the level of categorical perception, while the phonetic ABX is to evaluate phonetic representation, which doesn’t necessarily show whether the representations are categorical or not. This is because the decision is forced by the ABX task which only requires it to be closer than negative reference, not necessarily be tightly similar to positive reference as shown in Sylber (Figure 3). Also, phonetic ABX doesn’t have a densely sampled continuum of sounds which require a careful simulation to keep other aspects (e.g., pitch or duration) the same between pairs.

GPUs used for training (Q-2): For training each model of Sylber, CFM, and uLM on LibriSpeech, we used a single A6000 GPU with 48G RAM. For uLM on LibriLight, we used two of them.

HuBERT tokens with higher granularity (Q-3): Thank you for bringing this to our attention. We added resynthesis results using 500 and 2K K-means clusters by Tu Anh et al. (2023) [9], which use the 9th layer of HuBERT-Base model (Table 2 in the updated manuscript). However, we could not find a particularly new pattern by using higher granularity other than adding more refinement in intelligibility. The prosody or pitch reconstruction is still largely broken despite using the fine-grained 2K clusters. This result reiterates the finding by Tu Anh et al. (2023) [9], where they reported huge pitch errors in Hifi-GAN trained on those units, and the demo samples with unnatural pitch. We did not consider testing a continuous representation of HuBERT since that doesn’t have tokenization nor any compression in temporal dimension. The main reason for demonstrating the continuous version of Sylber is not for beating HuBERT, more for suggesting the upper bound limit of the syllabic tokenization of speech. Since Sylber shows very high temporal compression with high correspondence to syllables, the continuous tokens can be still useful and we believe knowing the full capacity of the representation is helpful. We leave a further investigation as a future work towards more effective quantization than the naive K-means clustering.

Detection and resynthesis performance long speech inputs (Q-4): We found minimal differences when we checked the performance with different input lengths longer than 5 seconds. The plots have been added to the revised manuscript in Figure 8 for your reference. As shown in the figure, the syllable detection scores have marginal differences across different input length bins. In terms of resynthesis, there is also minimal difference in input lengths longer than 5 seconds, slightly degrading in 15-20s and 20-25s inputs in resynthesis.

If you believe we have addressed the concerns, kindly upgrade your recommendation for this paper. Thank you

The necessity of syllabic tokens (W-7): In addition to the scalability issues previously raised, another question is whether syllabic tokens are necessary in real-world use cases. First, one of the main purposes of developing better speech tokenizers is to advance applications like ASR, TTS, and spoken LM. Nevertheless, the authors only presented a minor improvement in SLU. So, it is unclear whether Sylber helps downstream tasks. Second, because of the loss of fine-grained information (Table 9), tokens extracted from Sylber require incorporating separate speech encoders/tokenizers for more complex problems, which other SSL-based tokenizers or neural audio codecs could address. Perhaps providing convincing reasons and experimental evidence helps justify the need for syllabic tokens.

As we explained in the previous question, we achieved significant improvement in token length and coding efficiency with a consistent comparable performance with previous models that operate on inefficiently lengthy inputs. This reduction in length holds a strong potential in improving efficiency of other downstream models, especially those that utilize large Transformer architecture, where the complexity of the attention mechanism scales quadratically with sequence length. Besides, the scaling speech model has always been bottlenecked by the length of the inputs, compared to the text-based large model, requiring many GPUs [12, 16, 17, 18]. This is even challenging in an academic setting which doesn’t have enough GPU capacities to fit in long context inputs. Our work shows a great potential for resolving this problem.

For the second point, the loss of fine-grained information is prevalent in SSL-based tokenizers. This is well shown in our experiments on HuBERT, which loses way more prosodic information than Sylber as shown in Table 2. Therefore, it is common in SSL-based tokenization to incorporate separate speech encoders [13, 14, 15], for example, speaker embedding or pitch. However, this is actually a benefit, not a caveat. This means the speech information is disentangled and can be controlled/modeled independently. This disentanglement provides better interpretability and controllability of the model behavior, potentially improving safety and transparency of black-box deep models. It is indeed well known that neural audio codecs have fine-grained acoustic information. However, to our best knowledge, none of them is interpretable, often requiring complex model architectures to deal with multiple levels of codebooks, not to mention the significantly high computing cost due to high sampling rate [19, 20, 21]. These are caveats that existing neural audio codecs struggle with.

In fact, the categorical perception analysis (Figure 3) demonstrates that the discarded information is actually removing perceptually inconsequential variance, thereby symbolically abstracting the embedding. As a result, Sylber can achieve a significantly better coding efficiency than HuBERT.

As we are proposing a conceptually novel framework, we put our efforts more into demonstrating validity, efficiency, and quality of the proposed tokenization, rather than exhaustively investigating downstream applications.

Lastly, we would like to stress that our work also provides an interesting scientific insight that a syllabic, categorical representation may be a natural representation of speech itself. This is because in all processes from HuBERT to SDHuBERT and to Sylber, every step is self-supervised with minimal inductive bias, but syllables naturally emerge and do not degenerate. Moreover, we are the first to demonstrate that a machine categorical perception of speech arises just as in human perception. Beyond industrial interests/necessity/reasons, we believe these findings hold value in advancing our understanding of the nature of speech representation.

Actual inference efficiency (W-5): We thank the reviewer fo this suggestion. We evaluate the inference efficiency of Sylber and compare it to SDHubert to show the efficacy of using these models as segmentation models in the Table below. In the first two rows, we evaluate the RTF in the small batch size regime to measure the efficacy for real-time speech processing. We randomly sampled 32 LibriSpeech files and sequentially ran them through the model and measured the end-to-end latency in order to calculate the RTF. In the latter two rows, we evaluate the RTF in the large batch size regime to measure the efficacy for offline speech processing. We randomly sampled 32 batches of 32 files of LibriSpeech files and ran them through the model in a batched manner. All experiments used the same set of randomly selected files in the same order for both SDHubert and Sylber. Every experiment was run on a single A6000 GPU with 2 AMD EPYC 7513 32-Core Processor.

As we can see, Sylber’s linear time complexity is evident, as it is around 4x faster than SDHubert at segmenting speech.

Insignificant SLU improvements (W-6): The major gain of using Sylber tokens is in minimizing sequence length and bit-rate. To our best knowledge, speech coding frequency below 5 Tok/s has not yet been demonstrated in SLU. We further experimented with larger scale data to 66K hours of LibriLight. (The original models we reported were only trained on 960 hours of speech) We also added uLM with silence tokens “w/SIL” inserted in the gap between syllables more than 140ms. We were able to achieve a comparable sWUGGY to the model with similar resource settings (small model with cold-init in TWIST [12]). Furthermore, we achieve a sBLIMP score that is slightly better than the largest model with 13B parameters. This is a remarkable improvement given that the token length is below 5 per second and we used only a base small model with 125M parameters.

We also included prior models that use tokenizer based on HuBERT or variants, including the suggested NAST. However, we found SpeechTokenizer doesn’t report these metrics so we couldn’t include it.

The uLMs we presented in the original manuscript used LibriSpeech, 960 hours of data, for training. We explained this in Section 5.1: Datasets, which was mentioned altogether with other models which also used LibriSpeech. We now explicitly mention this specifically for uLM to avoid confusions, and the size of training data is added to Table 5. We updated the manuscript accordingly.

Complicated resynthesis approach (W-4): Our resynthesis model recruited a minimal setting for CFM and we found consistent patterns across different input types as shown in Table 2. In particular, we minimized complications as much as possible by removing input masking and context prompts which were utilized in previous CFM applications in audio or speech [10, 11]. Furthermore, we kept the size of the model moderately small as an 8 layers transformer, and the training configuration to be minimal as we needed to run many different models. Even the randomness is removed at inference to get deterministic output. The target outputs have only 14 dimensions which are significantly smaller than previous CFM applications. In the initial internal exploration phase, we found that a naive regression model that predicts articulatory features directly also works, however, we observed that CFM training is way faster than the regression model. Therefore, we settled down with using CFM for our resynthesis experiments as we need to train multiple models with our limited academic computation resources.

The previous studies brought up by the reviewer use Hifi-GANs, but those models also induce the known instability (e.g., mode collapse) in training GAN and complicated interplays with multiple losses. Therefore, we believe CFM would not specifically involve more complication than the previous methods. In fact, the pattern is highly consistent across HuBERT, SDHuBERT, and Sylber as shown in Table 2.

Regarding using Articulatory Encodec, the encoding and decoding pipeline is fixed and shared across different CFMs, and the previous study shows sufficiently high-performance of this model in terms of synthesizing speech. The decoder is a regular Hifi-GAN which is also utilized in the previous CFM applications where the acoustic features predicted by CFMs are fed to pretrained Hifi-GAN to synthesize audio. The main reason for adopting Articulatory Encodec is that the features are interpretable, so that we can have a better understanding of which and how speech information can be reconstructed from different speech tokens. Such interpretability is not available in other spectral features or VQ-VAE embeddings.

Regarding the performance, the intelligibility using Sylber is comparable to HuBERT units but the advantage is that Sylber units have significantly lower token lengths per second up to 7~8 times (or 3~4 times even when BPE applied to HuBERT). As shown in the coding-rate analysis in Table 6, Sylber shows a better trade-off in token lengths and intelligibility than HuBERT. Our listening tests for prosodic appropriateness reveal that, Sylber units show a superior prosody reconstruction with significantly higher pitch accuracy, making Sylber units potentially better at capturing suprasegmental information than HuBERT units. We now include the human subjective evaluation where listeners reported higher prosodic similarity in resynthesis using Sylber (psMOS).

We limited the quantization method to K-Means to make a fair comparison and to focus more on demonstrating preliminary results in this new concept of speech representation. However, a simple K-Means would not be an optimal way of quantizing syllable representations as syllables are combinatorial units of phonemes having onset, nucleus, and coda structure (in English). A future study will be conducted seeking a better quantization method that can reflect such structure and achieving the high-performance marked by resynthesis without quantization.

[1] Baevski et al. Data2vec: A general framework for self-supervised learning in speech, vision and language. ICLR 2022

[2] Cieri et al., The fisher corpus: A resource for the next generations of speech-to-text. In LREC, volume 4, pp. 69–71, 2004

[3] Pratap et al. Mls: A large-scale multilingual dataset for speech research. Interspeech 2020

[4] Shi et al. Aishell-3: A multi-speaker mandarin tts corpus and the baselines. Interspeech 2021

[5] Cho et al. Sd-hubert: Sentence-level self-distillation induces syllabic organization in hubert. ICASSP 2024

[6] McAuliffe et al. Montreal forced aligner: Trainable text-speech alignment using kaldi. Interspeech 2017

[7] Gorman https://github.com/kylebgorman/syllabify

[8] Sanz-Lázaro https://github.com/fsanzl/silabeador

[9] Nguyen, Tu Anh, et al. Expresso: A benchmark and analysis of discrete expressive speech resynthesis. arXiv 2023

[10] Le et al. Voicebox: Text-guided multilingual universal speech generation at scale. NIPS 2024

[11] Vyas et al. Audiobox: Unified audio generation with natural language prompts. arXiv 2023

[12] Hassid et al. Textually pretrained speech language models. NIPS 2024

[13] Kharitonov et al. Text-free prosody-aware generative spoken language modeling. ACL 2022

[14] Polyak et al. Speech resynthesis from discrete disentangled self-supervised representations. Interspeech 2021

[15] Nguyen et al. Spirit-lm: Interleaved spoken and written language model. arXiv preprint arXiv 2024

[16] Baevski et al. wav2vec 2.0: A framework for self-supervised learning of speech representations. NeurIPS 2020

[17] Hsu et al. Hubert: Self-supervised speech representation learning by masked prediction of hidden units. TASLP 2021

[18] Radford et al. Robust speech recognition via large-scale weak supervision. ICML 2023

[19] Wang et al. Neural codec language models are zero-shot text to speech synthesizers. arXiv 2023

[20] Chen et al. VALL-E 2: Neural Codec Language Models are Human Parity Zero-Shot Text to Speech Synthesizers. arXiv 2024

[21] Song et al. Ella-v: Stable neural codec language modeling with alignment-guided sequence reordering. arXiv 2024

Discussion on results in SUPERB (W-2): We present Sylber as more of a novel “coding” framework of speech grounded in a linguistic and phonological basis. Unlike general SSL models, many previous speech coding or codec studies rarely report SUPERB or diverse downstream performance. Instead, they put more focus on reconstruction quality and coding-rate given the targeted purpose of efficient coding of speech. While following the convention, we decided to include SUPERB as well since what we are presenting is conceptually very novel; therefore, we tried to include as much information as possible. However, given our major purpose and the limited space, we ended up putting the SUPERB results in the Appendix.

We acknowledge the fact that Sylber shows lower performance in some downstream tasks. However, the main purpose of Sylber is not in improving universal representation of speech, rather is more in interpretable representation with temporal structure, trading off some fine grained information of speech. As we show in the resynthesis results, the speech contents can be reliably restored from Sylber features, indicating that the lost information is more in non- or beyond-phonological abstractions learned in the original HuBERT. This may indicate that such lost abstractions are not compatible with syllable-like phonological abstraction, thus requiring another model on top of Sylber features. We believe such downstream models can be efficiently implemented as the input length can be significantly reduced in Sylber.

We strongly agree that the negative results should be equally presented. We are discussing this in detail in Appendix A.3. We would be happy to move this result to the main pages, but we are afraid that we are not able to do so due to the space limit and the aforementioned low priority of SUPERB experiments. Instead, we extended our discussion of our limitations in the main manuscript, and will take into account the discussion from these reviews into consideration in the final version of the paper.

Question on heat maps (Q-1): The heat maps are showing frame-level similarities, without any pooling or manipulation.

Robustness to initial segmentation (Q-2.a): We appreciate the reviewer for raising this very insightful question. We now include ablation experiments with different initial segments and models. We simulated a noisier setting by randomly adding noise to the initial segment boundaries. We randomly selected 20% of the boundaries and shifted 20-80ms, affecting 36% of the segment annotations in total. This perturbation is done once before the training. We then trained models with different feature model initialization by SDHuBERT or HuBERT. (We didn’t include training from scratch since the training was not successful.) Due to the limited time window, we trained the models with a reduced setting by decreasing EMA decay from 0.9995 to 0.999, increasing the learning rate from 1e-4 to 5e-4, and reducing the number of updates from 115K to 50K, and skipping the second stage training. We evaluated the syllable detection and discovery metrics.

As results, the models trained with the noisier initial segments are able to show high detection and discovery scores closer to the main Sylber model. The performance difference between two initializations is marginal and no single model outperforms in all metrics, indicating that HuBERT can be also used for feature model initialization for self-segmentation distillation. The frame-wise similarity matrices visualized in Figure 6 in the revised manuscript show similar prominent syllabic structures in both models.

We deeply appreciate the reviewer for their thoughtful evaluation of our paper and their constructive comments. We conducted additional experiments and analyses. Below, we address the raised weaknesses and questions in detail. We updated the manuscript accordingly.

Non-speech frames (W-1.1): We used non-speech frames obtained from segmentation algorithm proposed by SDHuBERT [1]. SDHuBERT demonstrated that frames near some syllable boundaries are “knocked out” at the later encoder layers, showing near-zero norm values. We leveraged these knocked out frames as non-speech frames by simply thresholding norms of frames following SDHuBERT. Additionally, we found many segments from SDHuBERT are non-speech, showing very low waveform amplitudes. Therefore, we filter out segments which have average absolute waveform amplitudes lower than 0.05 (calculated after z-scoring the entire waveform). This is explained in Appendix A.1.6: Threshold Setting in the revised manuscript. We added a short version of this explanation in the main Method section 3.1 and appended linkage to the Appendix with more detailed explanations. We thank the reviewer for bringing this to our attention.

Effect of clean representation of Sylber in segmentation algorithm (W-1.2): We thank the reviewer for suggesting this ablation experiment, which led to very interesting and useful findings. We evaluated syllable detection and discovery performance with different combinations of feature models and segmentation algorithms. When we apply the previous algorithm, MinCut, which was used by SDHuBERT, to Sylber, we find very marginal differences (Sylber-Mincut). MinCut is designed to search optimal segments at the cost of computation time. Therefore, this indicates that Sylber features are clean and robust enough to find optimal segments in a greedy manner. In fact, when the greedy algorithm is applied to SDHuBERT which is noisier than Sylber, we find significant drops from the original scores (SDHuBERT-Greedy), emphasizing the much cleaner embeddings in Sylber.

In the case of HuBERT, the differences are less prominent and mixed (HuBERT-Greedy), potentially due to the lack of syllabic structure in HuBERT features. We appended these results to the main Table 1 in the revised manuscript.

To share some more technical details, for applying greedy algorithms to other models, we searched for the optimal value of the hyperparameters on the dev-clean of LibriSpeech, maximizing R-value. Unlike Sylber features, we found that the greedy algorithm makes boundary decisions too frequently, severely over-segmenting the frames. To mitigate this, we add a preliminary merge operation before the local adjustment in our greedy algorithm, which merges a segment to the next segment if the segment average features of adjacent segments show high similarity. We use the same merge threshold parameter for determining high similarity. This new operation is also linear in time, and actually, it reduces the computation time by merging spurious boundaries and skipping the local adjustment.

I thank the authors for the very thorough responses. Theses additional experiments provide many interesting insight. Please include them in the manuscript. I have raised the score to 8.

We deeply appreciate the reviewer for recognizing our contributions and insights on the naturalness of resynthesis. We completely agree and have now conducted a human subjective evaluation to get MOS on naturalness (nMOS). And to be further informative, we also ask listeners to evaluate how naturally the prosody of resynthesized speech resembles the original. We denote this as prosodic similarity MOS or psMOS.

As result, we found superior naturalness in prosodic similarity in resynthesis from quantized units of our model compared to that from HuBERT units. The nMOS is similar in HuBERT with 2000 units and Sylber with 20K units. However, as the HuBERT units require significantly more bitrate, 370 bit/s, while ours only need 61 bit/s. We also find room for improvement when we compare to ours without quantization and the ground truth, which we will explore in future study. If you believe we have addressed the concerns, kindly upgrade your recommendation for this paper. Thank you.

Potential bias in metrics (Q-1): We'd like to point out that the entire process is self- or unsupervised from waveforms alone, with no “explicit” bias toward human annotated syllables. We are using unsupervised segmentation from SDHuBERT as initial targets for learning segmentation, and we used self-supervised segments from Sylber itself in the second stage training. As no information about the ground truth is given in any process, all the improvements we reported are self-driven, which applies the same for all the models compared. Therefore, we believe there is a minimal bias in the metrics.

Requirements of finer-grained representation (Q-2): As we are shifting from (sub) phonemic tokens to syllabic tokens, the token vocabulary requires a larger vocab size to cover frequently used combinations of phonemic units, to fully preserve intelligible contents. We believe that a sufficient level of intelligibility is a prerequisite for language understanding to emerge. Thus, a fine-grained cluster performs well since it can preserve contents more accurately. (But if length is taken into account, Sylber has much fewer granularity than previous HuBERT tokens as shown in Table 2 & 5). However, this may come with a trade-off as we need more training samples to mitigate a long tail issue with less frequent tokens. We expect a similar scaling law demonstrated in NLP literature can be shown using the proposed syllabic tokens since the temporal granularity is similar to that of subword tokens in text, allowing more efficient and effective scaling. In the current paper, we did not put much focus on exploring the scaling aspects, and kept the experiments in a minimal setting with a basic dataset. We leave this as an interesting direction for future work.

Structure in previous SSL models (Q-2): Indeed, we acknowledge that SSL models have sub-phonemic structures as previous studies have demonstrated. As we explain in the introduction, the ideal tokenization for speech has some desirable properties which can be beneficial: temporal segments and categorical discreteness, which are lacking in current tokenization schemes. As shown by [1, 2], the sub-phonemic structure in SSL would be highly continuous, densely tracing the contextualized or coarticulated phonetic representation. This continuous nature leads to inefficient coding in tokenization as we reported in Table 2 and 3.

[1] Sicherman & Adi. Analysing discrete self supervised speech representation for spoken language modeling. ICASSP 2023 [2] Cho et al. Evidence of vocal tract articulation in self-supervised learning of speech. ICASSP 2023

If you believe we have addressed the concerns, kindly upgrade your recommendation for this paper. Thank you

We deeply appreciate the reviewer for their thoughtful evaluation of our paper and their constructive comments. Below, we address the raised weaknesses and questions in detail.

Moderating claims in SLU improvements (W-1): We thank the reviewer for the suggestion. We agree with the reviewer’s point regarding our claims. In the revised manuscript, we toned down the statements by removing “better”, and for the second statement, we made it specific to be lexical and syntactic understanding rather than claiming gains in general SLU – thus leaving “syllabic units are suited for lexical and syntactic understanding”.

We also want to share the new experiments that can consolidate the moderated claim. We conducted more experiments with larger dataset (LibriLight 66K) and we found Sylber outperforms a huge size model in sBLIMP with significantly reduced token lengths, and is comparable in the sWUGGY metric with previous models with comparable model sizes (updated Table 5 in the revised manuscript). We also added uLM with silence tokens “w/SIL” inserted in the gap between syllables more than 140ms.

Also, we deeply appreciate the reviewer’s acknowledgement in efficiency and interpretability. Indeed, our major focus here is to present a novel framework, which is more relevant to “coding” studies for speech. With this perspective, we added visualization of the embedding space (Appendix A.7) to reinforce the interpretability of the Sylber embeddings. The plot highlights discreteness and phonological arrangement of proposed embeddings. Other main results we are presenting are largely focused on efficiency and quality of coding. We hope moderating those claims eases the concerns of the reviewer.

Adding details in Section 3.1 (W-2): We thank the reviewer for the suggestion. We added more details to Section 3.1 in the revised manuscript. Due to the space limit, many details of our approach and experiments are still placed in Appendix. We will optimize space better and make this section more comprhenesive by moving some of the experimental setup into the appendix.

Simplifying long sentences (W-3): We thank the reviewer for the suggestion. We will give a thorough check to avoid long sentences.

We greatly appreciate for dedicating your time and effort to reviewing our work. In our rebuttal and the revised paper, we have aimed to address all the concerns and suggestions raised in the initial reviews. We believe this has become a much stronger paper through the feedback and improvements suggested. We welcome further discussion on any of these points.

Additionally, we would like to share new interesting results we obtained while resolving other reviewers' comments. The follows are the most surprising results we found worth noting. These are all updated in the revised manuscript.

1. Zero-shot generalizability of syllabic segmentation to other domain and languages (Appendix A.5)
2. Stable training of Sylber even with the presence of noise in initial segment (Appendix A.6)
3. Additional human-evaluation by mean opinion score (MOS) revealing higher or similar subjective qualities of Sylber tokens compared to HuBERT based tokens (Table 4.)

If all your questions have been addressed and you find these new results interesting, we kindly ask you to consider updating your scores accordingly.

We sincerely appreciate your thoughtful feedback and support.