Response to Reviewer VGSo (1/2)

---

Reply to Weakness

W1: Since lexical tone decoding is not a very common task in the brain decoding area, I think the task / problem setting needs to be first introduced and defined in the methodology part.

R: We thank Reviewer VGSo for this suggestion. We will add the problem definition of tone decoding in the Method section in both the revised manuscript and the camera-ready version.

W2: The paper lacks analysis of the generalization ability of captured common representation. If the shared codebook of three subjects well generalizes to the fourth subject, whether any three of the four subjects lead to a similar shared codebook? The authors also mention this issue in limitations, but I think the dataset containing four subjects is enough to conduct such experiments.

R: As suggested, we conducted an additional ablation study to evaluate the generalization ability of the common representation captured by the public codebook. Specifically, we tested whether the public codebook, learned from three participants, could be generalized to a fourth participant. We followed the same experimental setup described in Sec. 4 with a leave-one-subject-out protocol.

Firstly, we applied the homogeneity-heterogeneity disentanglement (H2D) on three subjects to acquire a public codebook. Then, we pre-trained the fourth subject using this previously acquired public codebook while keeping it frozen and updating only the private codebook. Finally, we performed neural decoding (ND) on the fourth subject. The decoding performance was compared with baselines and variants of our proposed H2DiLR. The results are summarized in the table below:

The row labeled "Generalization Ablation" represents the ablation study, where we froze the public codebook after training it on three subjects and evaluated it on the fourth. We observe that even with the public codebook frozen for three subjects, decoding on the fourth subject still outperforms other baselines. In fact, the performance is better than our H2DiLR model trained with $\nu = 0$ and is comparable to the performance achieved when trained with UPaNT only. This confirms that the representation captured by the public codebook can be effectively applied to a new subject. Moreover, we noted that the performance improvement on Subject 2 was smaller compared to the other subjects. We hypothesize that this could be due to the distinct electrode implantation location for Subject 2, which likely differs from that of the other participants.

---

Reply to Question

Q1: The usage of non-invasive brain recordings is an emerging topic. What's the performance of H2DiLR in decoding EEG signals?

R: In addition to tone decoding using intracranial recordings, we demonstrated the applicability of H2DiLR for seizure prediction using non-invasive EEG signals (Appendix C.1). Beyond neurological signals, our model can be applied to general time-series data with a multi-channel setup, addressing data heterogeneity across different acquisition setups. This broader applicability extends the impact of our work to various domains within the community.

Response to Reviewer VGSo (2/2)

---

Q2:Why the size of the codebook and the dimensions of the codebook set the same?

R: The dimensions and size of the codebooks are flexible parameters, which we set to 256 by default, as this configuration offers an effective balance between reconstruction performance and the expressive capacity of the learned representations. A comprehensive ablation study on various codebook sizes and dimensions, presented in Table 3 of our manuscript, shows that the proposed approach maintains stable performance across different configurations, with the best results achieved using the default setting.

Thank you for your detailed rebuttal and answering all my points. Nevertheless, I still have some concerns:

Major

W1: OK. Perhaps the reason why “temporal jitter” doesn’t work is that you utilize the identified time stamps to epoch the data, which makes the feature wave time-locked to the onset. “additive noise” also doesn’t work,,, not properly preprocessed? Please see W2.

W2: So, no referencing preprocessing [1] is applied to the raw sEEG recordings? It’s important to note that the raw sEEG recordings are highly correlated (especially on the same electrode). Typically, this part needs to be removed by either laplacian reference [2] or bipolar reference [3,4] to obtain relatively cleaner signals that focus more on the activity of local neuronal groups. If the model directly reconstructs the sEEG recordings without referencing, the model will be biased towards the background noise of the brain (as the electrical signals spread out across the brain, like EEG). Could the authors explain this?

W3&W4: OK. Sharing the data and code would greatly enhance the validation and reproducibility of the proposed method by the research community. Please see Minor W5.

Minor:

W1: OK. Different parts of embedding sequence are indeed quantized by one of these two codebooks (public or subject-specific).

W2&W3&W4: OK.

W5: Thank you for providing the additional results. They are helpful, although the performance remains comparable to the (uni-modality) AAI model (26.78%) reported in [5]. I saw the results of seizure prediction, which is a channel-level classification task. I want to understand the effectiveness of H2DiLR on decoding cognitive states (e.g., speech decoding). Based on my understanding (which could be wrong), tune decoding is a subset of speech decoding, mainly involving speech motor brain regions (e.g., vSMC, STG). To my knowledge, Du-IN doesn’t address the issue of multi-subject modeling. Since the code and dataset for Du-IN [3] are publicly available, could H2DiLR outperform Du-IN? (optional, if there is enough time).

Question:

Q1: OK. K=32 is smaller than that used in Du-IN [3]. Considering Major W2, I’m not sure which part of the raw sEEG recordings the VQ-VAE focuses on. Could the authors explain this?

Q2&Q3: OK.

Summary:

The authors should provide more results to demonstrate the effectiveness of H2DiLR:

• The results after referencing preprocessing (e.g., bipolar re-reference [3,4]).
• If there is enough time, the results on the Du-IN dataset should be provided.

Reference:

[1] Li G, Jiang S, Paraskevopoulou S E, et al. Optimal referencing for stereo-electroencephalographic (SEEG) recordings[J]. NeuroImage, 2018, 183: 327-335.

[2] Wang C, Subramaniam V, Yaari A U, et al. BrainBERT: Self-supervised representation learning for intracranial recordings[J]. arXiv preprint arXiv:2302.14367, 2023.

[3] Zheng H, Wang H T, Jiang W B, et al. Du-IN: Discrete units-guided mask modeling for decoding speech from Intracranial Neural signals[J]. arXiv preprint arXiv:2405.11459, 2024.

[4] Mentzelopoulos G, Chatzipantazis E, Ramayya A G, et al. Neural decoding from stereotactic EEG: accounting for electrode variability across subjects[C]//The Thirty-eighth Annual Conference on Neural Information Processing Systems.

[5] Feng C, Cao L, Wu D, et al. A high-performance brain-sentence communication designed for logosyllabic language[J]. bioRxiv, 2023: 2023.11. 05.562313.

Response to Reviewer AheB (1/2)

---

Reply to Major Weakness

W1: The authors don’t mention the usage of data augmentation (e.g., temporal jittering, etc.) to avoid overfitting in sEEG-based speech decoding [2,3]. Since the number of available trials within each subject is too small, I’m not sure whether the improvement in pre-training comes from this.

R: As mentioned in the implementation details in Appendix B, we did not apply these data augmentations in the pre-training and fine-tuning stages. The effect of these augmentations can vary significantly depending on the hyperparameters, decoding tasks, and individual subjects [4]. In our experiments, we observed no consistent improvements across subjects when using the same set of augmentation hyperparameters, which is why we chose not to incorporate them in this work.

W2: The preprocess details are missing. Could you provide a detailed description of sEEG preprocessing (i.e., from the originally collected sEEG signals at a sample rate of 2kHz to the epoched trials at a sample rate of 1kHz)?

R: The neural signals were low-pass filtered at 300 Hz and notch filtered at 50 Hz, 100 Hz, 150 Hz, and 200 Hz to remove power line interference. The signals were then downsampled to 1000 Hz. It has been updated in the revised manuscript.

W3: The CNN baseline and the NAR model achieve about 40% and 45.75% on average in [1]. All these methods are supervised (heterogeneous) baselines, and these results make it hard to assess the contribution of H2DiLR.

R: As you mentioned, Feng et al. [1] utilized a multi-modal approach to combine both sEEG and audio data (as the multi-modality setting), whereas our proposed H2DiLR uses only sEEG data, making it a more generalizable framework. Building upon this difference, as reported in Table 4 in our manuscript, H2DiLR still achieves a higher accuracy (45.95%) compared to the results in [1], even with a slightly larger network structure and using sEEG data only.

W4: No code and demo dataset is provided. Without the code and data, it isn't easy to verify the claims.

R: We plan to release the code upon acceptance of this paper. As for the data, it is available from the corresponding authors upon reasonable request, ensuring compliance with ethical and privacy guidelines.

W5: Some aspects of the method are undefined or unclear. Please see the Questions section below. These aspects need to be clarified in the manuscript.

R: We have addressed the specific points you raised in the Reply to Question section. Please refer to our detailed responses for clarification, and we have ensured that the manuscript has been updated accordingly for improved clarity.

References

[1] Feng C, Cao L, Wu D, et al. A high-performance brain-sentence communication designed for logosyllabic language[J]. bioRxiv, 2023: 2023.11. 05.562313.

[2] Metzger S L, Littlejohn K T, Silva A B, et al. A high-performance neuroprosthesis for speech decoding and avatar control[J]. Nature, 2023, 620(7976): 1037-1046.

[3] Willett F R, Kunz E M, Fan C, et al. A high-performance speech neuroprosthesis[J]. Nature, 2023, 620(7976): 1031-1036.

[4] Wu, Di, et al. "Neuro-BERT: Rethinking Masked Autoencoding for Self-Supervised Neurological Pretraining." IEEE Journal of Biomedical and Health Informatics (2024).

---

Reply to Minor Weakness

W1: Figure 1 might mislead readers into thinking that H2DiLR uses Product Quantization. In Figure 3 and Section 3.3, the authors actually quantize each item in the embedding sequence using one of the different codebooks. However, “disentangled embedding” in Figure 1 suggests that different parts of each individual embedding are quantized with different codebooks.

R: As described in Sec. 3.3, we maintain a public codebook shared across all subjects and a subject-specific private codebook for each subject. Each neural embedding from an encoder is quantized using either the shared codebook or the subject-specific private codebook, based on a similarity ranking outlined in Equation 4. Thus, different parts of an individual embedding are indeed quantized by one of these two codebooks (public or subject-specific), which aligns with the depiction in Fig. 1. We believe this is consistent.

W2: Line 155: typo “LaBraM Anonymous (2024)”

R: It has been corrected in the revised manuscript.

W3: Line 841: typo “the learning of 5e-5” should be “the learning rate of 5e-5”? Is the only difference between fine-tuning pre-trained models and the non-pretrained baselines the learning rate? Are all other settings the same?

R: All other training settings during the ND stage are identical to those used for the non-pretrained baselines, except for the learning rate, which has been clarified in the revised manuscript. It has been corrected in the revised manuscript for clarity.

Our work targets a specialized subset of articulation decoding—tone decoding—which specifically involves decoding articulatory properties rather than semantic content. In contrast, Du-IN addresses word-level decoding, which focuses possibly more on semantics rather than articulatory.

Major W2

To my knowledge (which could be wrong), speech decoding mainly involves speech motor information. This information is concentrated in certain brain areas [1,2,3]. Please see Figure 2d in [2] for ECoG-based channel contribution analysis. sEEG recordings are different from ECoG [2] and MEA [1,3]. They don't have to select channels, as they already have ECoG and MEA in that brain region. That’s why I recommend the authors to perform top-k channel selection before downstream analysis. All aforementioned works [4-7] including perform these operation, e.g., BrainBERT for listening decoding [4], Du-IN for speech decoding [5], etc.

Besides, bipolar (or laplacian) re-reference is commonly used to explore the brain encoding problems in neuroscience [8,9]. That’s also why I recommend the authors to do bipolar re-reference before downstream neuroscience analysis and decoding tasks.

For example,

• in [8], The preprocessing began by converting the iEEG signals to bipolar derivations by pairing adjacent electrode contacts. Recording sites in the hippocampus were paired with a nearby white-matter electrode that was identified anatomically using FreeSurfer’s segmentation (68).
• in [9], We used bipolar rather than unipolar derivations because it allows for better signal artifact removal and achieves high spatial resolution (~3 mm) by canceling out contributions of distant sources which spread equally to recording sites (39).

Shouldn't the data be properly preprocessed first to ensure the algorithm achieves optimal performance?

Minor W5

However, the primary goal of masked modeling and VQ-VAE-based pre-training is not subject alignment but rather learning generalized features that can be effectively fine-tuned for downstream tasks. Altough the shared tokenization and reconstruction process does not directly enforce alignment across subjects but facilitates the encoder's ability to extract task-relevant and transferable features.

Du-IN [3] reconstructs 10 channels within certain brain regions with codex_size=2048, which is greatly larger than K=32 used for the whole brain recordings reconstruction (e.g., >50 channels). Are you sure the bandwidth of invasive sEEG recordings is that low? -- with 32 discrete codes and a conv-deconv autoencoder, you can reconstruct the signals of the whole brain, overlooking the brain's desynchronization nature.

My concern mainly focuses on element-wise alignment. Only 4 subjects are used to pre-train H2DiLR, why not involving more subjects from Brain Treebank dataset [6] and Du-IN dataset [5]. I hope that H2DiLR can achieve higher performance after introducing more subjects. H2DiLR($\nu=0$) with K=32 within 1 subject is much smaller than K=256 used in H2DiLR(UPaNT+H2D) across 4 subjects. This discrepancy compromises the fairness of the comparison and may underestimate the performance of the (heterogeneous) H2DiLR($\nu=0$) baseline.

Reference:

[1] Card N S, Wairagkar M, Iacobacci C, et al. An accurate and rapidly calibrating speech neuroprosthesis[J]. New England Journal of Medicine, 2024, 391(7): 609-618.

[2] Metzger S L, Littlejohn K T, Silva A B, et al. A high-performance neuroprosthesis for speech decoding and avatar control[J]. Nature, 2023, 620(7976): 1037-1046.

[3] Willett F R, Kunz E M, Fan C, et al. A high-performance speech neuroprosthesis[J]. Nature, 2023, 620(7976): 1031-1036.

[4] Wang C, Subramaniam V, Yaari A U, et al. BrainBERT: Self-supervised representation learning for intracranial recordings[J]. arXiv preprint arXiv:2302.14367, 2023.

[5] Zheng H, Wang H T, Jiang W B, et al. Du-IN: Discrete units-guided mask modeling for decoding speech from Intracranial Neural signals[J]. arXiv preprint arXiv:2405.11459, 2024.

[6] Wang C, Yaari A U, Singh A K, et al. Brain Treebank: Large-scale intracranial recordings from naturalistic language stimuli[C]//The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track. 2024.

[7] Mentzelopoulos G, Chatzipantazis E, Ramayya A G, et al. Neural decoding from stereotactic EEG: accounting for electrode variability across subjects[C]//The Thirty-eighth Annual Conference on Neural Information Processing Systems.

[8] Norman Y, Yeagle E M, Khuvis S, et al. Hippocampal sharp-wave ripples linked to visual episodic recollection in humans[J]. Science, 2019, 365(6454): eaax1030.

[9] Domenech P, Rheims S, Koechlin E. Neural mechanisms resolving exploitation-exploration dilemmas in the medial prefrontal cortex[J]. Science, 2020, 369(6507): eabb0184.

R: We plan to release the code upon acceptance of this paper. As for the data, it is available from the corresponding authors upon reasonable request, ensuring compliance with ethical and privacy guidelines.

Major

W2

At least, you have to ensure that the pre-training of H2DiLR ($\nu=0$) indeed converges; my major concern still focuses on whether H2DiLR ($\nu=0$) with K=32 can successfully reconstruct the >50-channel sEEG signals. Besides, after the bipolar re-reference, you should at least select top-k channels based on specific brain regions or encoding results [1,2,3,4], bi-polar reference only provides cleaner signals that focus on the activity of local neuronal groups.

Sharing the data and code would greatly enhance the validation and reproducibility of the proposed method by the research community.

Reference:

[1] Wang C, Subramaniam V, Yaari A U, et al. BrainBERT: Self-supervised representation learning for intracranial recordings[J]. arXiv preprint arXiv:2302.14367, 2023.

[2] Zheng H, Wang H T, Jiang W B, et al. Du-IN: Discrete units-guided mask modeling for decoding speech from Intracranial Neural signals[J]. arXiv preprint arXiv:2405.11459, 2024.

[3] Wang C, Yaari A U, Singh A K, et al. Brain Treebank: Large-scale intracranial recordings from naturalistic language stimuli[C]//The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track. 2024.

[4] Mentzelopoulos G, Chatzipantazis E, Ramayya A G, et al. Neural decoding from stereotactic EEG: accounting for electrode variability across subjects[C]//The Thirty-eighth Annual Conference on Neural Information Processing Systems.

Further Response to Reviewer AheB

---

Reply to Major Weakness

W2: It’s important to note that the raw sEEG recordings are highly correlated (especially on the same electrode). Typically, this part needs to be removed by either laplacian reference [2] or bipolar reference [3,4] to obtain relatively cleaner signals that focus more on the activity of local neuronal groups. Based on the aforementioned decoding studies and my experience with sEEG decoding, bipolar (or laplacian) reference greatly improves decoding performance. If you believe H2DiLR still works after bipolar re-reference, please do that ablation.

R: Thank you for your feedback and suggestions. As noted in our previous response, Laplacian and bipolar referencing are not universally required preprocessing steps, and many recent studies, including [6,7,8], have employed common referencing strategies similar to ours.

That said, to address Reviewer AheB's concern, we conducted additional experiments using bipolar referencing (computing differences between nearest contacts on the same electrode). Following the exact same experimental protocol, we observed results comparable to those obtained with our previously adopted common referencing approach:

This suggests that the choice of referencing technique has minimal impact on decoding performance in our work. We believe this further supports the notion that referencing methods depend on factors such as acquisition hardware, inter-subject variability, and decoding task requirements, rather than adhering to a universal standard.

---

Reply to Minor Weakness

W5-1: So why not a larger codebook for H2DiLR ($\nu=0$), e.g., 256?

R: The main contribution of our work is to propose a novel Homo-heterogeneity Disentanglement learning paradigm in addition to existing Heterogeneous training and Homogeneous training. We reported the perforamnce of H2DiLR (UPaNT only) and H2DiLR ($\nu=0$) only for reference purpose and to provide a more comprehensive comparison to other baselines. As our proposed H2DiLR utilizes a codebook size of 32 for private codebooks, we set the K=32 for H2DiLR ($\nu=0$) to ensure a fair comparison.

W5-2: I have indicated that VQ-VAE only has the power to align different subjects space-wise (instead of element-wise). Yeah, all subjects use the same set of tokens, but you introduce different stems (tokenize raw sEEG signals into tokens) and different heads (reconstruct raw sEEG signals from tokens). This design cannot encourage element-wise alignment; how could this pre-training across subjects work? For example, due to the lack of element-wise alignment, a possible situation is that the sEEG signals from different subjects sharing the same meanings belong to different trajectories on the same sphere.

R: Based on our understanding, we assume that "space-wise" alignment refers to token-level alignment, where tokens represent the quantized representations. And "element-wise" alignment likely refers to instance-level alignment, where each instance corresponds to the full neural representation composed of multiple tokens.

Reviewer AheB is concerned that the UPaNT pre-training approach might not sufficiently align representations across subjects to produce a clustering effect. We acknowledge that autoencoding-based pre-training methods such as VQ-VAE and masked autoencoders typically do not achieve the clustering effects often seen with contrastive learning approaches. Contrastive learning excels at producing representations with high linear separability across subjects, even without fine-tuning.

However, the primary goal of masked modeling and VQ-VAE-based pre-training is not subject alignment but rather learning generalized features that can be effectively fine-tuned for downstream tasks. Altough the shared tokenization and reconstruction process does not directly enforce alignment across subjects but facilitates the encoder's ability to extract task-relevant and transferable features.

The performance improvements we observe during fine-tuning suggest that this generalization successfully supports downstream tasks, even if it does not explicitly produce tightly clustered subject-specific representations.

Thank you for your detailed rebuttal and answering all my points.

Major

W2

It’s important to note that the raw sEEG recordings are highly correlated (especially on the same electrode). Typically, this part needs to be removed by either laplacian reference [2] or bipolar reference [3,4] to obtain relatively cleaner signals that focus more on the activity of local neuronal groups.

Core Concern: Based on the aforementioned decoding studies and my experience with sEEG decoding, bipolar (or laplacian) reference greatly improves decoding performance. If you believe H2DiLR still works after bipolar re-reference, please do that ablation.

Minor

W5

H2DiLR (UPaNT only) employs a larger codebook size (K=128) compared to H2DiLR ($\nu=0$), which uses K=32. A larger codebook provides more capacity for learning diverse representations, which could explain why H2DiLR (UPaNT only) achieves better performance.

So why not a larger codebook for H2DiLR ($\nu=0$), e.g., 256?

I have indicated that VQ-VAE only has the power to align different subjects space-wise (instead of element-wise). Yeah, all subjects use the same set of tokens, but you introduce different stems (tokenize raw sEEG signals into tokens) and different heads (reconstruct raw sEEG signals from tokens). This design cannot encourage element-wise alignment; how could this pre-training across subjects work? For example, due to the lack of element-wise alignment, a possible situation is that the sEEG signals from different subjects sharing the same meanings belong to different trajectories on the same sphere.

Question

Q1

Subcortical areas such as the Thalamus also play a significant role in tone decoding

(optional) The result of this contribution value is interesting. Is there any relevant neuroscience research showing that these brain regions encode speech-motor information?

Reference

[1] Li G, Jiang S, Paraskevopoulou S E, et al. Optimal referencing for stereo-electroencephalographic (SEEG) recordings[J]. NeuroImage, 2018, 183: 327-335.

[2] Wang C, Subramaniam V, Yaari A U, et al. BrainBERT: Self-supervised representation learning for intracranial recordings[J]. arXiv preprint arXiv:2302.14367, 2023.

[3] Zheng H, Wang H T, Jiang W B, et al. Du-IN: Discrete units-guided mask modeling for decoding speech from Intracranial Neural signals[J]. arXiv preprint arXiv:2405.11459, 2024.

[4] Mentzelopoulos G, Chatzipantazis E, Ramayya A G, et al. Neural decoding from stereotactic EEG: accounting for electrode variability across subjects[C]//The Thirty-eighth Annual Conference on Neural Information Processing Systems.

Further Response to Reviewer AheB

---

Reply to Major Weakness

W2: This operation is the standard configuration of sEEG recordings. So, the aforementioned concerns still exist -- no laplacian [2] or bipolar [3,4] re-reference is applied to remove the highly correlated parts across channels (e.g., the noise within certain brain regions). Please refer to [1] for the detailed definition of bipolar re-reference. For example, if 15 channels are located on one electrode (e.g., A1, ..., A15), bipolar re-reference will calculate the differences between nearby neighbors (e.g., A1-A2, A2-A3, ..., A14-A15). Du-IN [3] reconstructs 10 channels within certain brain regions with codex_size=2048, which is greatly larger than K=32 used for the whole brain recordings reconstruction (e.g., >50 channels). Are you sure the bandwidth of invasive sEEG recordings is that low? -- with 32 discrete codes and a conv-deconv autoencoder, you can reconstruct the signals of the whole brain, overlooking the brain's desynchronization nature [5].

R: In our work, the primary focus is on extracting task-relevant decoding features rather than explicitly removing noise within certain brain regions. It is worth emphasizing that there is no universally accepted definition of "noise" in functional brain regions, nor is there a gold standard for referencing in the domain of EEG or sEEG measurements. While Laplacian or bipolar referencing is a common practice, it is not mandatory. Many recent and impactful studies have employed a common referencing strategy similar to ours. For instance:

• Huang et al. (2023) in Nature Communications [7]: "We re-referenced the signal from each channel/contact to the average signal across all the white matter channels."
• Regev et al. (2024) in Nature Human Behaviour [8]: "The common average reference (from all electrodes connected to the same amplifier) was removed for each time point separately."
• Fan et al. (2024) in Nature Mental Health [9]: "The continuous intracranial EEG data in each electrode contact was notch-filtered at 60 Hz, re-referenced to the common average reference, and segmented from -500 ms to 1,000 ms relative to stimulus onset."

These examples highlight that the referencing strategy we used is well-established and accepted in the field.

Regarding the codebook size, we would like to clarify that codebook usage efficiency is equally critical as the codebook size itself. Large codebooks often suffer from low utilization rates, meaning many codes remain unused during training. As we stated earlier, our approach utilizes a total of 256 codes, with 128 in the public codebook and 32 per subject-specific private codebook. Our experiments confirmed a high code usage rate, which supports effective learning and reconstruction. As shown in Figure 6 and Table 3 of our paper, our approach achieves strong reconstruction performance both quantitatively and qualitatively, validating our chosen codebook configuration.

Furthermore, the reconstruction task serves as a pretext task in VQ-VAE training to enable effective representation learning. While high reconstruction accuracy is important, the ultimate goal is to ensure the learned representations are generalizable and task-relevant. We hope this clarification demonstrates the rationale behind our referencing strategy and codebook design, which balance the needs of task-relevant feature extraction and efficient model training.

---

Reply to Minor Weakness

W5-1: This is just optional. However, their codes have already provided the list of 10 channels for each subject to reproduce the results (with the corresponding checkpoints). Compared to Feng et. al. [6], H2DiLR (26.31%) performs similarly to the supervised (heterogeneous) AAI model (uni-modality, 26.78%). On tune decoding, they also perform similarly (45.95%(H2DiLR)v.s.[40%(CNN),45.75%(NAR)]), although the authors clarify that 45.75% is obtained by a (multi-modal) NAR model. All these results make it hard to assess the effectiveness of H2DiLR.

R: NAR is a multi-modal method that integrates both audio and sEEG data. In contrast, our proposed H2DiLR achieves a performance of 45.95% using only sEEG data, surpassing NAR's 45.75%, which benefits from additional audio information. This demonstrates H2DiLR's capability to extract meaningful representations from sEEG data alone, without relying on external modalities.

Similarly, the AAI model leverages initial category features as additional supervision, introducing external guidance beyond the sEEG signal. Despite this, our approach demonstrates comparable performance on decoding tasks when restricted to using sEEG data alone. This comparison underscores H2DiLR's effectiveness as a standalone decoding framework, even when directly compared to methods utilizing auxiliary inputs or supervision.

Major

W2

OK. This operation is the standard configuration of sEEG recordings. So, the aforementioned concerns still exist -- no laplacian [2] or bipolar [3,4] re-reference is applied to remove the highly correlated parts across channels (e.g., the noise within certain brain regions). Please refer to [1] for the detailed definition of bipolar re-reference. For example, if 15 channels are located on one electrode (e.g., A1, ..., A15), bipolar re-reference will calculate the differences between nearby neighbors (e.g., A1-A2, A2-A3, ..., A14-A15). Du-IN [3] reconstructs 10 channels within certain brain regions with codex_size=2048, which is greatly larger than K=32 used for the whole brain recordings reconstruction (e.g., >50 channels). Are you sure the bandwidth of invasive sEEG recordings is that low? -- with 32 discrete codes and a conv-deconv autoencoder, you can reconstruct the signals of the whole brain, overlooking the brain's desynchronization nature [5].

Minor

W5

This is just optional. However, their codes have already provided the list of 10 channels for each subject to reproduce the results (with the corresponding checkpoints). Compared to Feng et. al. [6], H2DiLR (26.31%) performs similarly to the supervised (heterogeneous) AAI model (uni-modality, 26.78%). On tune decoding, they also perform similarly (45.95%(H2DiLR)v.s.[40%(CNN),45.75%(NAR)]), although the authors clarify that 45.75% is obtained by a (multi-modal) NAR model. All these results make it hard to assess the effectiveness of H2DiLR.

In Du-IN, Du-IN (mae) corresponds to H2DiLR ($\nu=0$), Du-IN (poms) corresponds to H2DiLR (UPaNT only). When they built the cross-subject model, they encountered the problem of representation conflict. Their Du-IN (mae) (i.e., H2DiLR ($\nu=0$)) achieves higher performance compared to Du-IN (poms) (i.e., H2DiLR (UPaNT only)). VQ-VAE has the power to align different subjects space-wise (instead of element-wise). How could H2DiLR ($\nu=0$) perform worse than H2DiLR (UPaNT only)?

Additional

Our analysis indicates that the Precentral and Postcentral Gyri are key contributors among cortical regions, with subcortical regions like the Thalamus also playing a significant role in tone decoding.

Besides, I still have an additional concern about the neuroscience ablation results. As you mentioned,

Our work targets a specialized subset of articulation decoding—tone decoding—which specifically involves decoding articulatory properties rather than semantic content. In contrast, Du-IN addresses word-level decoding, which focuses possibly more on semantics rather than articulatory.

Considering Major W2, could the authors provide some neuroscience research to support this result?

Summary: My major concern mainly focuses on

• whether the data has been properly preprocessed, this is crucial for the downstream analysis and decoding performance.
• the effectiveness of cross-subject performance.

Reference:

[1] Li G, Jiang S, Paraskevopoulou S E, et al. Optimal referencing for stereo-electroencephalographic (SEEG) recordings[J]. NeuroImage, 2018, 183: 327-335.

[2] Wang C, Subramaniam V, Yaari A U, et al. BrainBERT: Self-supervised representation learning for intracranial recordings[J]. arXiv preprint arXiv:2302.14367, 2023.

[3] Zheng H, Wang H T, Jiang W B, et al. Du-IN: Discrete units-guided mask modeling for decoding speech from Intracranial Neural signals[J]. arXiv preprint arXiv:2405.11459, 2024.

[4] Mentzelopoulos G, Chatzipantazis E, Ramayya A G, et al. Neural decoding from stereotactic EEG: accounting for electrode variability across subjects[C]//The Thirty-eighth Annual Conference on Neural Information Processing Systems.

[5] Buzsaki G. Rhythms of the Brain[M]. Oxford university press, 2006.

[6] Feng C, Cao L, Wu D, et al. A high-performance brain-sentence communication designed for logosyllabic language[J]. bioRxiv, 2023: 2023.11. 05.562313.

Response to Reviewer AheB (2/2)

---

Reply to Minor Weakness

W5: Thank you for providing the additional results. They are helpful, although the performance remains comparable to the (uni-modality) AAI model (26.78%) reported in [5]. I saw the results of seizure prediction, which is a channel-level classification task. I want to understand the effectiveness of H2DiLR on decoding cognitive states (e.g., speech decoding). Based on my understanding (which could be wrong), tune decoding is a subset of speech decoding, mainly involving speech motor brain regions (e.g., vSMC, STG). To my knowledge, Du-IN doesn’t address the issue of multi-subject modeling. Since the code and dataset for Du-IN [3] are publicly available, could H2DiLR outperform Du-IN? (optional, if there is enough time).

R: It’s important to note that speech prostheses and current speech decoding systems [6-7] often focus on decoding motor movements associated with speech production, such as lip or tongue movement during phoneme articulation. Our work targets a specialized subset of articulation decoding—tone decoding—which specifically involves decoding articulatory properties rather than semantic content. In contrast, Du-IN addresses word-level decoding, which focuses possibly more on semantics rather than articulatory. This distinction introduces variability in the neural states being decoded and necessitates different approaches for channel selection.

Specifically, Du-IN employs a data-driven channel selection strategy, choosing the channels that contribute most to the decoding task during training. This approach inherently biases the system toward the specific decoding algorithm/network architecture used, as evidenced by the significant performance differences between Du-IN (~ 60%) and other methods like Brant [8] (~ 10%) and LaBraM [9] (~10%). Authors of Du-IN highlight the substantial contributions of white matter in their decoding tasks, especially for Subjects 2 and 3, as shown in Tab. 17. Conversely, our work focuses on decoding speech-related neural signals, explicitly excluding white matter channels to align with our tone decoding task.

While Du-IN’s dataset is publicly available, its accompanying channel localization details (beyond the ten selected channels) are not fully provided. Since channel selection rules critically influence results, directly comparing our method to Du-IN without access to equivalent channel configurations would not result in a fair or unbiased evaluation.

That said, we’ve already demonstrated the generalizability and effectiveness of H2DiLR on various tasks, including tone decoding, initial syllable decoding, and seizure prediction. This versatility underscores its potential as a general-purpose framework for neural decoding. While it would be ideal to evaluate H2DiLR on more tasks like word-level decoding, resource and time constraints prevent us from exhaustively addressing all potential decoding scenarios within the scope of this work.

Nevertheless, we acknowledge the importance of exploring the DU-IN dataset for a more comprehensive comparison and will make our best effort to analyze it within the constraints of time and available information. We appreciate your suggestion and will aim to provide insights, if feasible, into how H2DiLR performs on this dataset.

---

Reply to Question

Q1: K=32 is smaller than that used in Du-IN [3]. Considering Major W2, I’m not sure which part of the raw sEEG recordings the VQ-VAE focuses on. Could the authors explain this?

R: Our H2DiLR framework employs a public codebook size of 128 and a private codebook size of 32 for each subject. With four subjects, the total number of codes utilized in the H2D stage amounts to 256.

The variant of H2DiLR with $\nu=0$ represents a special case that simplifies the framework to focus solely on personalized representation, allowing for broader comparison with other methods.

As outlined in Equation 4, each neural embedding derived from the raw sEEG signal is quantized using either the public codebook or the subject-specific private codebook. The selection process is guided by a similarity ranking mechanism, enabling the model to effectively capture both homogeneous and heterogeneous features within the data.

Thanks for your contribution to the field of sEEG-based tone decoding. I hope that if your paper is accepted, you will make the code and pre-processed data publicly available, just like Du-IN [1], which will ensure the reproducibility of the core claims -- H2DiLR pre-training on 4 subjects yields better representations than pre-training on 1 subject.

---

In my Summary Comment by Reviewer AheB, I expressed two primary concerns:

• One pertains to the neuroscience analysis (Major W2),
• The other relates to the effectiveness of H2DiLR (Minor W5),

I'm not sure whether ICLR requires strict standards for neuroscience analysis. During the rebuttal stage, I noticed an unfair configuration in the (heterogeneous) H2DiLR($\nu=0$) baseline when compared to H2DiLR(UPaNT+H2D), i.e., H2DiLR($\nu=0$) with K=32 within 1 subject is much smaller than K=256 used in H2DiLR(UPaNT+H2D) across 4 subjects. This discrepancy compromises the fairness of the comparison and may underestimate the performance of the (heterogeneous) H2DiLR($\nu=0$) baseline.

I suggested an ablation study to address this issue:

W5-1: So why not a larger codebook for H2DiLR ($\nu=0$), e.g., 256?

Unfortunately, the authors declined to provide this result, stating:

The main contribution of our work is to propose a novel Homo-heterogeneity Disentanglement learning paradigm in addition to existing Heterogeneous training and Homogeneous training. We reported the perforamnce of H2DiLR (UPaNT only) and H2DiLR ($\nu=0$) only for reference purpose and to provide a more comprehensive comparison to other baselines. As our proposed H2DiLR utilizes a codebook size of 32 for private codebooks, we set the K=32 for H2DiLR ($\nu=0$) to ensure a fair comparison.

Based on my understanding (which could be wrong), with K=32 discrete codes and a conv-deconv autoencoder, reconstructing the signals of the whole brain (>50 channels; 4 hours) is unlikely to succeed. Prior studies support this:

• VQ-VAE applied to sEEG with 10 channels (15 hours) used a codebook of K=2048 [1];
• VQ-VAE applied to EEG with 128 channels (12 hours) also employed K=2048 [2].

I also requested the authors to share their code and a demo dataset to verify the claim that -- H2DiLR pre-training on 4 subjects yields better representations than pre-training on 1 subject. However, the authors declined to provide these materials. Without them, I am unable to validate this crucial assertion, which directly affects the evaluation of H2DiLR's effectiveness. Besides, the authors are also recommended to do another ablation, your conv+xfmr with raw sEEG signals as input (with data augmentation, bi-polar re-reference, top-k selection, and commonly used trun_normal(0., 0.02) weight initialization in Transformer). Because the time-domain sEEG signals itself are already a kind of representation that is very easy to distinguish based on the bias introduced by conv. After all, in the field of EEG and sEEG, few people use VQ code for representation, such as LaBraM [3], VQ-MTM [4], and Du-IN [1]. Because of these concerns, I still think that the ablation experiment of this work is not satisfactory, which is why I didn't raise my score.

Additionally, the authors and I agreed that -- VQ-VAE only has the power to align different subjects space-wise (instead of element-wise). The authors subsequently reframed their claims:

However, the primary goal of masked modeling and VQ-VAE-based pre-training is not subject alignment but rather learning generalized features that can be effectively fine-tuned for downstream tasks. Altough the shared tokenization and reconstruction process does not directly enforce alignment across subjects but facilitates the encoder's ability to extract task-relevant and transferable features.

To the best of my knowledge (which could be wrong), pre-training on only 4 subjects for H2DiLR(UPaNT+H2D) does not seem sufficient to achieve the reported improvements over H2DiLR($\nu=0$) (trained on 1 subject). For this reason, I suggested including additional subjects from resources like Brain Treebank and Du-IN. Importantly, I did not request evaluations on new tasks.

Reference:

[1] Zheng H, Wang H T, Jiang W B, et al. Du-IN: Discrete units-guided mask modeling for decoding speech from Intracranial Neural signals[J]. arXiv preprint arXiv:2405.11459, 2024.

[2] Duan Y, Zhou J, Wang Z, et al. Dewave: Discrete eeg waves encoding for brain dynamics to text translation[J]. arXiv preprint arXiv:2309.14030, 2023.

[3] Jiang W B, Zhao L M, Lu B L. Large brain model for learning generic representations with tremendous EEG data in BCI[J]. arXiv preprint arXiv:2405.18765, 2024.

[4] Gui H, Li X, Chen X. Vector quantization pretraining for eeg time series with random projection and phase alignment[C]//International Conference on Machine Learning. PMLR, 2024: 16731-16750.

Response to Reviewer ScW5

---

Reply to Weakness

W1: The authors limited themselves somewhat in specializing their approach to tone decoding only. I feel that the problem of disentangling subject-specific effects from general ones is a wider problem with many downstream applications, but also for problems of encoding as well. A deeper exploration of this is expected in the work.

R: Indeed, our proposed H2DiLR is a general-purpose framework for homogeneity-heterogeneity disentangled neural representation learning, capable of being applied to various neural decoding tasks. In this work, we focused on tone decoding as a proof of concept to validate the framework's effectiveness.

Furthermore, to demonstrate the versatility of H2DiLR beyond tone decoding, we applied it to a seizure prediction task using non-invasive EEG signals. The results, detailed in Appendix C.1, further illustrate the framework's adaptability to diverse neural decoding applications.

W2 and Q1: There is also disappointingly little exploration of the abundant neuroscience questions here - I would like to have seen a location-based ablation in addition to the parametric ablations of the method that showed which regions are most useful for tonal decoding. An analysis studying if there are region-specific differences to representational homogeneity.

R: Due to the dual-stage training paradigm of our H2DiLR framework and the challenges of differentiating through quantization during backpropagation, we cannot directly analyze the contributions of specific regions during neural decoding.

To address this, we conducted a separate analysis by combining the homogeneity-heterogeneity disentanglement (H2D) pre-training and neural decoding stages into a single-stage model. We applied cross-entropy loss with a Transformer decoder to quantized representations, using a straight-through estimator (STE) to approximate gradients. Please note, however, that this setup may introduce training instability and slight performance degradation; it is used solely to analyze region contributions.

We calculated channel saliency scores through class activation maps, normalized between 0 and 1, and aggregated them across brain regions linked to speech processing (e.g., Superior Temporal Gyrus, Precentral Gyrus, Postcentral Gyrus), as well as reference regions like the Angular Gyrus. Our findings are summarized below:

Our analysis indicates that the Precentral and Postcentral Gyri are key contributors among cortical regions, with subcortical regions like the Thalamus also playing a significant role in tone decoding.

---

Reply to Question

Q2: Is there any benefit of this method for neural encoding models that predict neural responses from feature input?

R: Thanks for your question. In encoding tasks, while different subjects may produce distinct neural responses to the same stimuli due to physiological differences (heterogeneity), similar neural patterns can still emerge across individuals as a result of shared encoding mechanisms (homogeneity). By disentangling these heterogeneous and homogeneous components, H2DiLR can capture both individual-specific and shared neural response characteristics, potentially enhancing the model's ability to generalize across subjects in encoding tasks.

Response to Reviewer mtfq (1/3)

---

Reply to Weakness

W1: The statement that a "comprehensive set of 407 Mandarin syllables covers nearly all Mandarin characters" requires further clarification, e.g., providing the specific vocabulary list.

R: Mandarin syllables consist of an initial and a final, and to ensure comprehensive coverage of all possible pronunciations with their corresponding tones, our reading material includes all potential combinations of initials and finals. Due to space constraints, we have included the complete vocabulary list in Tables A4-A7 of the revised manuscript, and we have highlighted the relevant entries in red for ease of reference.

W2: The concept of private and shared codebooks is introduced in the paper but lacks a detailed explanation. A more comprehensive rationale and description are needed to clarify how and why these codebooks are defined and utilized.

R: The intuition behind our proposed H2DiLR is straightforward: although physiological and instrumental differences exist among different subjects (heterogeneity), the same brain regions in different individuals exhibit similar functions during the tone production process (homogeneity).

Within H2DiLR, we implement this balance through two types of codebooks. The shared codebook stores homogeneous information accessible to all subjects and captures the commonalities across individuals, while subject-specific private codebooks store heterogeneous information unique to each subject. Section 3.3 introduces the structure and role of these codebooks.

The mathematical definition and training process for homogeneity-heterogeneity disentanglement (H2D) is detailed in Section 3.4, with a visual illustration provided in Figure 3. Specifically, neural representations from each encoder are quantized through either the shared or private codebook based on a similarity ranking outlined in Equation 4. The updating rules for the shared and private codebooks are given in Equations 4 and 5, respectively, while the overall loss functions for encoder updates are detailed in Equation 6 and Equation 7.

W3: Sharing the data and code would greatly enhance the validation and reproducibility of the proposed method by the research community. This transparency would contribute significantly to the field and support further advancements.

R: We plan to release the code upon acceptance of this paper. As for the data, it is available from the corresponding authors upon reasonable request, ensuring compliance with ethical and privacy guidelines.

W4: The claim of being "the first" to work with 407 Mandarin syllables is inaccurate, as the 2023 study titled "A high-performance brain-to-sentence decoder for logosyllabic language" also used a dataset with 407 syllables. Although this work is cited, the related work section lacks a detailed discussion of this study.

R: Thanks for mentioning the relevant work. Although our study and the 2023 work by Feng et al. utilize the same dataset setting with 407 syllables, we are indeed the first to perform unified tone decoding across subjects rather than the first to use the 407 syllables setting. Moreover, the objectives and approaches differ significantly. Feng et al. focused on developing a sentence-level decoding system tailored to each individual subject, employing a heterogeneous training paradigm. In contrast, our work aims to improve tone decoding performance by disentangling and learning both heterogeneous and homogeneous neural representations across all subjects. This approach, which we refer to as Homo-heterogeneity Disentanglement, allows us to capture shared and distinct neural patterns more effectively without increasing model complexity per subject.

Response to Reviewer mtfq (2/3)

---

W5: The paper asserts that it disentangles homogeneity and heterogeneity, yet the evidence provided relies on feature visualization of specific samples. A statistically robust analysis is necessary to substantiate this claim.

R: The visualization result verifies that the public codebooks capture homogeneity in terms of tone decoding. We also conducted another subject classification task to verify whether the heterogeneous representations learned by our proposed H2D capture the heterogeneity. This task shows that the learned heterogeneous representation extracts sufficiently discriminative information to classify sEEG signals from different subjects. We conducted an additional ablation study to evaluate the generalization ability of the homogeneity captured by the public codebook. Specifically, we tested whether the public codebook, learned from three participants, could be generalized to a fourth participant. We followed the same experimental setup described in Sec. 4 with a leave-one-subject-out protocol.

Firstly, we applied the homogeneity-heterogeneity disentanglement (H2D) on three subjects to acquire a public codebook. Then, we pre-trained the fourth subject using this previously acquired public codebook while keeping it frozen and updating only the private codebook. Finally, we performed neural decoding (ND) on the fourth subject. The decoding performance was compared with baselines and variants of our proposed H2DiLR. The results are summarized in the table below:

As shown in the row labeled "Generalization Ablation" in the ablation table, we froze the public codebook after pre-training it on three subjects and evaluated it on the fourth. We observe that even with the public codebook frozen for three subjects, decoding on the fourth subject still outperforms other baselines. In fact, the performance is better than our H2DiLR model trained with $\nu = 0$ and is comparable to the performance achieved when trained with UPaNT only. This confirms that the representation captured by the public codebook can be effectively applied to a new subject.

Additionally, we noted that the performance improvement on Subject 2 was smaller compared to the other subjects. We hypothesize that this could be due to the distinct electrode implantation location for Subject 2, which likely differs from that of the other participants.

---

Reply to Question

Q1: Can you provide the specific list of 407 Mandarin syllables and explain how they represent nearly all Mandarin characters?

R: We have included the full vocabulary list in Tables A4-A7 of the revised manuscript, and we have highlighted the relevant entries in red for ease of reference.

Q5: You utilize a ConvNet with 1.55M as the encoder and a Transformer with 0.958M as the neural decoder for each subject. This implies you implement joint training for multiple subjects in a multi-task learning manner. A perfect cross-subject paradigm should not require additional parameters and should support cross-subject testing, which your design evidently does not. The drawback is that with each additional subject, the total number of parameters increases further.

R: Thanks for raising this critical point. Our model explicitly targets the heterogeneity introduced by both physiological and instrumental factors in invasive brain neural decoding. For example, due to variations in subject conditions, different participants have varying numbers of implanted electrodes, resulting in differences in channel counts across subjects. This variability presents a unique challenge that distinguishes our work from tasks like EEG decoding, where all subjects typically share a standardized electrode configuration (e.g., the 10-20 system).

Given these differences, designing a universal decoder capable of handling the variability in electrode configurations across subjects is inherently complex. While our current approach employs subject-specific decoders to address this challenge effectively, we fully acknowledge that a universal network capable of supporting cross-subject testing without increasing the number of parameters per subject would be an ideal solution.

We are actively exploring methods to develop a single encoder-decoder framework that can accommodate all subjects, aiming to improve efficiency and scalability in future iterations of our work. We appreciate the reviewer's insight and agree that achieving such a universal architecture is an important direction for further research. However, this does not diminish the contribution of our proposed homogeneity-heterogeneity disentanglement (H2D) learning paradigm, which offers a novel and effective approach to tackling a fundamental challenge in neural decoding.

---

Q6:Regarding the definition of private and shared codebooks, what is the total number of sEEG channels across multiple subjects? I am confused about the private codebook with K=32?

R: The sEEG channel counts for the participants in our study are A, B, C, and D, respectively. Each subject’s data is represented as $\mathcal{X}_{i} \in \mathbb{R}^{N_i \times T \times C_i}$, where $N_i$ is the number of data samples, $C_i$ is the number of channels (which varies across subjects), and $T$ is the segment length, which is consistent for all subjects.

To address this heterogeneity in channel counts, neural recordings $\mathcal{X}$ are first mapped into a latent feature space, $z = E_{i}(x; \theta_i) \in \mathbb{R}^{L \times D}$, using a set of subject-specific encoders ${E_{i}(\cdot; \theta_i)}_{i=1}^m$. Here, $L$ is the sequence length of tokens in the latent space, and $D$ is the dimensionality of each token.

For quantization, we employ a public codebook with $K=128$ and private codebooks with $K=32$ learnable key-value pairs, represented as $\mathcal{C} = {(k, e(k))}_{k \in [K]}$, where each code $k$ has an associated learnable embedding $e(k) \in \mathbb{R}^{D}$. Each token in $z$ is discretized into a code using the quantization function described in Equation 1 of our paper. This discretization process is crucial for capturing both subject-specific (private) and shared features across subjects, enabling the disentanglement of heterogeneity and homogeneity.

The private codebook with $K=32$ is specifically designed to handle subject-specific variations in neural recordings, ensuring effective encoding of private information. By contrast, the shared codebook is responsible for modeling similarities across subjects. Together, they enable our homogeneity-heterogeneity disentanglement (H2D) approach to address the variability in channel counts and individual differences.

References

[1] Si, Xiaopeng, et al. "Cooperative cortical network for categorical processing of Chinese lexical tone." Proceedings of the National Academy of Sciences (2017): 12303-12308.

[2] Oganian, Yulia, et al. "A speech envelope landmark for syllable encoding in human superior temporal gyrus." Science advances (2019): eaay6279.

[3] Lu, Junfeng, et al. "Neural control of lexical tone production in human laryngeal motor cortex." Nature Communications (2023): 6917.

---

Thank you for your detailed feedback. We believe our clarifications address your concerns and strengthen the technical rigor of our work. We hope this work can reach a broader audience within the community, as we feel it tackles an important challenge.

Your evaluation is highly valuable to us, and we respectfully invite you to consider increasing your rating. We welcome any further questions or points for clarification.

We look forward to your feedback and deeply appreciate your consideration.

Warm regards,

Authors

Reply to Question

---

Q1: I am curious about how you determined your vocabulary list and what dependencies it involved. You consistently emphasize in sections A4-A7, "The first/second/... set of 407 Mandarin Chinese characters used as reading material." My understanding is that the combined vocabulary lists of the four subjects should cover the total number of syllables.

R: Thanks for the inspiring comment, and your understanding is correct. The syllables presented in Tables A4-A7 indeed total 407. During our experiments, each participant read carrying sentences containing a complete set of all 407 target syllables three times. Each Mandarin syllable is composed of an initial, a final, and a tone, which collectively define its pronunciation. The reading material used in the study was designed by Mandarin linguists to ensure comprehensive coverage of all possible pronunciations, along with their corresponding tones. While Mandarin has a vast lexicon of over 50,000 characters, its pronunciations map to one of these 407 unique syllables, each paired with one of the four tones. This design guarantees that the reading material is sufficient to represent the full spectrum of Mandarin pronunciation, making it an optimal choice for our decoding task.

---

Q2: A critical issue: your introduction is quite confusing, and it is unclear what exactly you are doing after reading it. Is the EEG segment your input of a specific length, and does this EEG signal correspond to the reading time? Most importantly, are you performing a classification task or a regression task? If it is a classification task, how many categories are there? If it is a regression task, are you decoding speech end-to-end, or are you only decoding text? I believe that instead of overemphasizing what you consider your innovation, "homogeneity-heterogeneity separation learning," it would be more beneficial to outline the specific tasks and details of your work clearly.

R: Thank you for your detailed feedback. However, we respectfully disagree with the assertion that our introduction is unclear. In this section, we structured our narrative as follows:

• Background and Importance: We introduced the linguistic background of tonal languages such as Mandarin, highlighting the critical role of tone in semantic interpretation and its importance for accurate brain-to-text decoding (lines 32-48).

• Limitations of Prior Work: We reviewed existing tone-decoding approaches using both non-invasive and invasive neural recordings, identifying key challenges, particularly the heterogeneity caused by physiological and instrumental factors (lines 49-60).

• Motivation and Framework: We presented the motivation for our study and an overview of our proposed H2DiLR framework, accompanied by Figure 1, to illustrate its structure and purpose (lines 61-90).

• Contributions and Results: A brief summary of our experimental setup, key findings, and contributions were provided (lines 91-109).

While we acknowledge the reviewer's concerns, we believe that the technical specifics of the task and experimental design are better suited to the Methods and Experiments sections, where they are thoroughly explained. To directly address the reviewer's specific questions:

• Q2.1: Is the EEG segment your input of a specific length, and does this EEG signal correspond to the reading time?

$\quad$ R: As described in Appendix A, the input sEEG signals are fixed-length 1-second segments aligned with the syllable reading process.

• Q2.2: Are you performing a classification task or a regression task? If it is a classification task, how many categories are there? If it is a regression task, are you decoding speech end-to-end, or are you only decoding text?

$\quad$ R: The decoding task in the second stage is a four-class classification task corresponding to the four tones of Mandarin syllables.

We appreciate the reviewer's suggestion and will revise the introduction to ensure clearer presentation, particularly highlighting the nature of the task earlier in the narrative.

Dear Reviewer mtfq,

As the review deadline of December 2 approaches, we wanted to kindly follow up regarding your feedback on our response to your questions.

We would greatly appreciate it if you could confirm whether all your concerns have been resolved. Should you feel our response merits reconsideration of your evaluation score, we kindly request that you update your score to reflect this.

Thank you again for your time and effort in reviewing our work.

Best regards,

Authors