Resolving Domain Shift For Representations Of Speech In Non-Invasive Brain Recordings

Weaknesses:

Utility - It seems as if the utility of non-invasive decoding studies as a whole is being called into question, though please do clarify your original statement if this is being misinterpreted. Given the high practical and ethical thresholds for surgery, there is certainly a space for studies that are not dependent on invasive techniques. Non-invasive decoding problems are well established in the literature as valuable scientific contributions. As just one example, see the paper by Défossez et al., whose architecture is one of the focuses of this work.

Approach Justification - The call for a comparison with other DA methods would certainly strengthen the paper and is a valuable suggestion. As we have responded to other reviewers, these other methods have yet to be applied to MEG and there remains a substantive difference between statistical transformations in sensor space and the kind of feature-level DA we employ here.

Clarity - We are sorry that you feel the paper is poorly written and would appreciate any suggestions you might have as to what specifically you felt was confusing or unnecessary. With respect to the achieved advancements, we would like to underscore that while the improvements over the original baselines (i.e. architectures training and evaluating on the same dataset) may seem incremental they are (a) statistically significant and (b) much larger improvements when comparing the performance pooling across studies with DA to that of the original architectures when pooling without. We acknowledge that we could do more to emphasize this last point and will make edits to the relevant sections of the paper accordingly.

Data Management - Results for the experiments with the Brainmagick base architecture are reported for both subsets using 15% of the data as well as the datasets in their entirety. While these datasets may seem small in comparison to other domains, this study is engaging with a volume of MEG data larger than the vast majority of other studies in the field (which usually use a single MEG study such as Gwilliams, alone). The experiments conducted with the MEGalodon base architecture were restricted to only the subsets, but it is reasonable to believe the scaling effect demonstrated in the Brainmagick case would hold here as well.

Impact - The primary objective of this paper is to demonstrate the viability of feature-level DA as a path towards integrating more MEG data sources for training a single architecture, and would like to re-emphasize the points made in response to “Clarity”. While we acknowledge your arguments with respect to the age-related analysis, those findings are secondary to the principle impact of this work.

Questions:

With respect to the first question, to which set of experiments are you referring? For Brainmagick, if referring to Table 2 the “Pre-trained on A” simply means that the training of the architecture on B was initialized from the final saved weights of training the baseline on A, as opposed to a random initialization. For MEGalodon, the framework has a pre-training stage and a fine-tuning stage where the dataset in the fine-tuning stage is never used during pre-training. A flow diagram is a good suggestion and could be added to further support the information included in sections A.4 and A.5 of the appendix.

The primary metric highlighted by the original authors is top-10 accuracy, whose convention we follow. For the same reason as Defossez et al. we did not want to spend unnecessary space reporting the top-1 accuracy, though we acknowledge your point and can include those results in the appendix upon revision.

All confidence intervals are computed over 3 seeds as reported in Table 2’s caption. Computational and time constraints prevented additional seeds from being collected for table 3.

"We do not find that the base Brainmagick architecture is effective in cross-dataset generalization" means that the original architecture, without augmentation for DA, has a decline in performance when pooling datasets for training. In other words, without the augmentations applied in this work the original architecture is unable to generalize over multiple studies to improve decoding performance. These results are provided in Table 2.

Reducing parameterization was not the focus of our study, but rather DA and improving cross-dataset generalization. As a result we sought to maintain the two original architectures we augmented as they were proposed. Interesting point though and we will consider this in the future.

Thank you for your responses, and I am just following up to gather feedback for future iterations of this work. With respect to Top-1 accuracy, I have uploaded a revised copy of this paper with the table included in the appendix (as Défossez et al. did in their paper) and will be included the results at the bottom of this comment. As stated before, these results are in line with what we found for the top-10 accuracy for the full dataset testing and subset testing.

I agree that the paper would be strengthened by additional DA comparisons, though I am still somewhat unclear as to why you feel that the use of current SOTA decoding models as the baselines to be augmented for DA is a worse choice than augmenting a simpler architecture. Both references you include (the second of which is not a peer reviewed paper) for SPOC, while used with MEG data, are not examples of an application to a speech decoding task. I feel it may be worth clarifying at this time, as I have also done in the revised copy of this paper, that the speech detection task for MEGalodon in this work is not the identification of the onset of speech from resting state MEG but rather the detection of natural breaks between words from an audio stimulus of continuous speech. This is a significantly more challenging task than classifying the onset of auditory stimulus.

Weaknesses:

We strived to be as transparent as possible with our methods. If you could share which specific aspects of the methodology you were unclear on, these details can be highlighted in further edits. We also acknowledge that the primary contribution of our work is their application to a previously unexplored modality and the related insights with respect to MEG DA, rather than proposing an entirely new framework. The objective was not to introduce a novel theoretical approach to DA.

Questions:

This is a good suggestion and such a formatting would have been a stronger visualization of the effect being shown. Unfortunately, the model was only checkpointed every 100 epochs, meaning we cannot generate such a series of graphics at this time. We opted instead with Figure 4 to contrast the probability distributions before and after DA. Would it be more illustrative to provide a contrast with the probability distribution after 200 epochs without applying DA as well?

Weaknesses:

A comparison with DA methods for EEG was omitted at this time as the current focus was on MEG, a modality that has been largely overlooked with respect to DA thus far. Granted, EEG and MEG do share many similarities but they remain distinct data sources.

Questions:

All confidence intervals and statistical power analyses were conducted with three seeds per assay. The key contributions of the paper are a novel application of feature-level DA to MEG data. This is important as this approach promotes minimal feature engineering of the raw data, avoiding potential sources of bias or accidentally removed signal. While this framework has been used before for computer vision and 3D neuroimaging data (MRI), this is the first application of it to time-series neuroimaging data and demonstrates its potential as an architecture-agnostic approach towards pooling multiple studies. In a functional sense, we also contribute findings supporting the importance of optimizer choice with respect to ADDA frameworks (e.g. improvements using SGD over ADAM for the adversarial head). Other DA methods for time-series neuroimaging data (EEG) are established in the literature, but are grounded in statistical transformations of the input data rather than harmonization at the model level.

Weaknesses:

1 - Good suggestions which we will take into account.

2 - Subject level differences were not addressed because each of the architectures examined already achieved inter-subject generalizability (within a single study). This was done with a subject-specific embedding for MEGalodon and a subject-specific layer for Brainmagick

3 - The results section for MEGalodon and the discussion addresses the discrepancy between speech detection and voicing and explains why it is an artifact of the original framework’s training protocol. The focus should be on the improvement achieved in voicing classification, the “deep fine-tuning” case. We maintained the original frameworks of the two base architectures we used in order to be as fair as possible while benchmarking. Can you provide more explanation behind the comment on the performance being related to the number of epochs? All benchmark comparisons are trained under the same conditions.

Questions:

This is a reasonable suggestion, though we would like to highlight that one of the benefits of our chosen approach is that by operating in feature space one could apply the demonstrated framework on top of any sensor space transformations (such as Zhang et al.’s MCCA) to likely yield further improvements. We acknowledge the potential concern that the performance gains could be marginal, however, as previous DA studies have failed to demonstrate inter-study generalization with MEG data we maintain that our contributions here are impactful to the field.