MAD: Multi-Alignment MEG-to-Text Decoding

The primary concern is the limited dataset, as the experiments were only conducted on the GWilliams dataset, while the model was trained with MEG-Speech-Text modality pairs. Testing on a single dataset limits the model's robustness and generalizability claims.

There appears to be some issue with citation formatting that should be fixed (see ICLR 2025 style guide), as well as a few grammatical issues. Additionally, the benchmarking comparison with Défossez et al. (2023) appears to be in bad faith. Instead of comparing the performance of their decoding framework against Défossez et al.'s model as originally designed, it seems as if the authors of this paper use only the brain model and then apply a decoding head in the style of their framework. Thus, it results in many of the generated unigrams being gibberish as opposed to actual words and greatly reduces whatever the real performance of the Défossez et al. model should be across all reported metrics. The original model from Défossez et al. returns the most likely segment of audio given the meg input and therefore always returns real words. The paper by Défossez et al. explicitly mentions the ability to decode segments not present in the training set. Thus, it is still reasonable to benchmark with the Défossez et al. model as originally designed. If the intention was to underscore that the model from Défossez et al. needs access to the segmented test audio while the proposed framework operates with MEG data alone, this could have been stated and the Défossez et al. model omitted from bechmarking comparison.

Originality

• Brain activity (invasive and non-invasive) to speech/text translation has already been introduced in prior works, as summarized in section 2 (related work).

• The methodological contribution is incremental. Défossez et al. (2023) introduced the idea of aligning M/EEG signals with latent representations of a pre-trained ASR model (wav2vec2.0) with the CLIP loss. In the submission, the authors use a related approach (whisper encoder instead of wav2vce2.0, and MMD loss instead of CLIP). Yang et al. (2024) combined the whisper model with a convolutional adapter network (trained with AdaLoRa). In this submission, the authors propose a modified architecture that uses the brain module of (Défossez et al. 2023) as adapter network.

Quality

• While I appreciate that the authors report results of the considered performance metrics for random effects (i.e., random shuffling and noise as input), their approach is insufficient to actually estimate the significance of the results. Instead of a single evaluation they should have used permutation testing (see e.g., Maris 2012) to estimate the metrics' distribution under the null hypothesis (i.e., no-relation between the MEG-derived text and the ground truth).

• Unlike (Défossez et al. 2023) and (Yang et al. 2024) the authors decided to analyze only one public dataset. I think the authors should have also analyzed the results for the (Schoffelen et al. 2019) dataset. Beyond that, the results lack quantification of the stability/variability of the results across random weight initializations.

Clarity

• Figure 1(a) does not correspond to the proposed MAD model in Table 2. This discrepancy suggests that the study might be affected by the double dipping problem (Kriegeskorte et al. 2009) in the sense that the authors tested many different configurations and then picked the final MAD model based on the test set result.

References

A. Défossez, C. Caucheteux, J. Rapin, O. Kabeli, and J.-R. King, “Decoding speech perception from non-invasive brain recordings,” Nat Mach Intell, Oct. 2023, doi: 10.1038/s42256-023-00714-5.

Y. Yang et al., “NeuSpeech: Decode Neural signal as Speech,” Jun. 03, 2024, arXiv: 2403.01748.[Online]. Available: http://arxiv.org/abs/2403.01748

J.-M. Schoffelen, R. Oostenveld, N. H. L. Lam, J. Uddén, A. Hultén, and P. Hagoort, “A 204-subject multimodal neuroimaging dataset to study language processing,” Sci Data, vol. 6, no. 1, p. 17, Apr. 2019, doi: 10.1038/s41597-019-0020-y.

E. Maris, “Statistical testing in electrophysiological studies,” Psychophysiology, vol. 49, no. 4, pp. 549–565, 2012, doi: 10.1111/j.1469-8986.2011.01320.x.

N. Kriegeskorte, W. K. Simmons, P. S. F. Bellgowan, and C. I. Baker, “Circular analysis in systems neuroscience: the dangers of double dipping,” Nat Neurosci, vol. 12, no. 5, pp. 535–540, May 2009, doi: 10.1038/nn.2303.

• Dataset: The reliance on a single dataset reduces the impact of the results. Although the authors claim good performance on entirely unseen text, the word overlap is 46% on stories belonging to the same corpus and same subjects. This assumption should be validated across other datasets.

• Losses: The authors indicate that the Le loss is the most important in the study, but they do not explain why. A part-of-speech analysis could clarify the performance differences between content and function words. They assert that high-level speech is better decoded than low-level speech; however, this has been demonstrated in previous studies (https://www.biorxiv.org/content/10.1101/2024.04.19.590280v1). Additionally, the authors do not provide a solid hypothesis for why the Lt loss negatively impacts the BLEU-1 score. They also do not show the performance when only using Lt, as they do for Lm and Le.

• Noise Study: In the Evaluation Metrics section, noise is employed to demonstrate that the model effectively learns from MEG signals. It would be more insightful to train the model on noise rather than simply evaluating it.

• Metrics: Although the paper mentions five different metrics, the discussion predominantly focuses on the BLEU-1 score, while the ROUGE-1 and CER scores exhibit different (opposite?) behaviors that do not show SOTA results. Notably, one of the main findings regarding the detrimental effect of Lt is not corroborated by the CER metrics. While it is commendable to use five metrics, the authors primarily discuss BLEU-1 without explaining why it is the most relevant metric.

• Remarks:

  • Introduction and Related Work: Consider integrating the related work section into the introduction for better flow, or relocating some introductory elements to the related work section.
  • Dataset Description: The paper states 2 hours per session, but it should be corrected to 1 hour per session, as per the MEG-MASC paper (Methods/Participants section, https://www.nature.com/articles/s41597-023-02752-5).
  • Limitations: The limitations section is underdeveloped and should be expanded.
  • Typos: Correct "setted" to "set" (section 4.1), "primarized" to "prioritized" (section 4.4.1), add a missing space between "state-of-the-art" and "(SOTA)" (conclusion), and address incorrect references (Supplementary A1).
  • References: On two different computers, the PDF displays a "double" reference in the Related Work section, mentioning the same studies twice in succession.

The residual connection in Figure 1 (b) is missing an arrow. The main experimental results in Table 2 have no advantage over RS and NeuSpeech, and are almost all lower except for B-1 and self-B. The brain module within Wav2vecCTC is not trained on text, so it exhibits poor performance and struggles to generate coherent words. The comparison is unfair. Baseline models should be compared both w/ and w/o tf. The paper aims to showcase the model's ability to generalize to unseen text, yet this attribute is not evident within the experimental setup presented. The paper solely conducts experiments on a test set with 46% overlap in words, which fails to represent unseen text. For a more thorough evaluation, it is crucial to compare the model with the baselines on both seen and unseen text separately. Furthermore, experiments should be conducted on additional datasets to enhance the validation of generalization. Ablation studies have demonstrated that L_t and LoRA are redundant and should be omitted from the model structure.