We sincerely appreciate Reviewer 1WBp for your questions and suggestions. Below are our responses. (W and Q stand for Weakness and Question, respectively.)

[W1 W2 Q1] Architecture Figure and Code

Thanks very much for your great suggestion. At this moment, NeurIPS does not permit the inclusion of rich media or external links during the rebuttal phase this year. We have prepared an architectural figure and will include it in the final version to make the paper easy to follow. In addition, we would be happy to share relevant portions of our code in Markdown format during the discussion phase if you have concerns regarding reproducibility.

[Q2] ViT-based Encoder Justification

Our choice of a ViT-based encoder for fMRI is motivated by both practical and representational considerations:

• Global Context Modeling: Our fMRI inputs are flattened into 1D vectors, which discards the original spatial layout of voxels. Unlike CNNs, which rely on local receptive fields, Vision Transformers (ViTs) have global receptive fields, enabling them to model long-range dependencies across brain regions. This helps compensate for the loss of spatial structure in our preprocessed inputs.
• Modality Alignment with CLIP: Since the visual features we use are extracted from a pretrained ViT-B/16 CLIP model, using a ViT-based encoder for fMRI improves architectural compatibility and facilitates cross-attention fusion. This design choice reduces the modality gap and encourages more effective alignment between neural and visual representations.

[Q3] More Datasets

We appreciate your comment on the generalization of the data set. We agree that demonstrating cross-dataset generalizability is an important step toward establishing a more comprehensive and robust framework.

We introduce subject identification as an auxiliary task to support our main motivation: understanding the neural mechanisms of visual processing in a data-driven manner. This focus requires datasets that include paired natural visual stimuli and corresponding neural responses across multiple subjects. Given this requirement, only a few datasets are suitable—namely, NSD, HCP Movie, and BOLD5000.

Among them, NSD is the only one that provides a reproducible and well-documented benchmark for semantic object classification tasks. In contrast, both HCP Movie and BOLD5000 lack established benchmarks for classification-based decoding, and existing studies primarily focus on reconstruction or resting-state analysis. Moreover, models trained on these datasets often do not release pre-trained weights, and the corresponding preprocessing pipelines are either unavailable or not well-documented, making fair comparisons infeasible.

We recognize the value of cross-dataset evaluation and plan to extend our experiments to HCP and BOLD5000 in future work. We will include this clarification in the final version of the manuscript.

[Q4] Text Modality

Thank you for raising this point. We chose not to include the text modality in our current model for both practical and methodological reasons.

• Motivational Misalignment: Our primary goal is to study semantic decoding from vision-related brain activity, using only visual cortex voxels. In the inference stage, it is not practical to assume text data is available for test samples. Introducing text features risks semantic leakage, as object names often appear explicitly in captions (e.g., in COCO descriptions), which could bias the model to cheat on text rather than decoding semantic content from fMRI. This would compromise our central objective of probing the brain's visual decoding mechanisms.

• Implementation Complexity: The CLIP visual features used in our model are extracted from the image encoder prior to text-image alignment, meaning they do not share a common latent space with CLIP text embeddings. To incorporate text features, we would need to retrieve aligned image-text pairs and re-extract features from the jointly aligned CLIP embedding space. Additionally, the visual embeddings before and after alignment differ in dimensionality, which introduces incompatibility with our current architecture. It is also important to note that CLIP aligns global image and text embeddings, whereas our model operates on local visual features for fine-grained semantic decoding. Incorporating text would therefore require substantial changes to our input representation and necessitate retraining the entire model—including the cross-attention module and downstream components—along with extensive hyperparameter tuning. Due to time constraints during the review period, we were unable to carry out this significant modification.

However, we consider your suggestions to be a promising direction for investigating brain-language functionalities in which we are very intereste and will certainly explore them in our future work.

We thank Reviewer 6F63 for your constructive feedback. Below are our responses. (W and Q stand for Weakness and Question, respectively.)

[W1] Semantic Decoding and Feature-level Interpretability

We appreciate your insightful comments on the limitations of existing approaches and the challenges of interpretability in object-level neural decoding.

Many recent works in neural decoding have prioritized generating high-fidelity visual reconstructions from fMRI data. While these outputs are visually impressive, such methods often rely heavily on pretrained generative models and focus primarily on image realism rather than understanding the neural signals themselves. As a result, they contribute marginally to advancing our knowledge of how the brain encodes visual information. This fundamental gap motivates our work: rather than asking what the image looks like, we ask how semantic object representations are encoded and how they vary across individuals at the neural level.

In our work, we take a discriminative decoding approach that simultaneously predicts object categories (semantic decoding) and subject identities (biometric decoding) from fMRI signals. We agree that object classification alone does not fully capture the richness of neural representations. However, we argue that our framework provides a meaningful step toward interpretability by enabling structured, task-relevant decomposition of neural information.

Our strong performance in semantic decoding allows us to leverage post-hoc interpretability tools (i.e., GradCAM, Attention Rollout) to analyze both model behavior and brain function. Specifically:

• Voxel-object activation maps (Fig. 2–3) and subject-specific attention analyses (Fig. 4–6) uncover both shared and individualized patterns of neural activity in response to visual stimuli；
• These analyses reveal attention dynamics and voxel selectivity that offer a data-driven perspective on how the brain represents object-level semantics—an aspect typically overlooked in reconstruction-based methods;
• By combining subject-object disentanglement with voxel-level interpretation, we address a largely underexplored question: how consistent are object-specific neural encodings across subjects, and where do individual differences emerge?

[W2] Variance Pushed to the Null Space

We appreciate this observation. It is indeed true that our disentanglement and decoding objectives prioritize task-relevant variance—specifically, for subject identification and object classification—which may lead to the *omission of other latent dimensions not directly aligned with these tasks. This trade-off is inherent to discriminative frameworks that aim for linear separability in service of specific objectives.

As clarified in Section 2.3 and Appendix A.2, our method is designed to isolate linearly separable components of neural representations relevant to subject and object decoding, rather than capturing the entire variance present in the fMRI signal. It is therefore expected that residual information—such as low-variance components or task-irrelevant patterns—may reside in the orthogonal null space of the learned latent basis.

We view this as a valuable opportunity rather than a limitation. In fact, one of the longer-term goals of our framework is to first isolate task-relevant components in a structured, interpretable manner, and then iteratively explore the null space to investigate what information remains. This could include:

• Unsupervised or contrastive objectives to identify structure in residual variance;
• Temporal dynamics or contextual dependencies not captured by static object categories;
• Subtle cognitive or emotional correlates that are not labeled but may still be encoded in the data; -Or even noise patterns that correlate with session-specific or scanner-specific artifacts.

We believe this two-stage process—(1) identifying interpretable, task-relevant features and (2) probing the remaining space—is a promising direction toward more transparent, comprehensive, and human-aligned neural decoding models, and we plan to pursue this in future work.

Regarding your concern about transparency at the latent feature level: we strongly agree that interpretability of individual latent dimensions remains a fundamental challenge in deep learning regardless of tasks or modalities. Our method does not claim to assign direct semantic meaning to each latent unit. Instead, we probe the structure of the latent space through aggregate behavioral and voxel-wise analyses. This work provides a framework that moves toward more interpretable models, and we view detailed latent-space analysis (e.g., basis axis meaning, feature attribution, spectral structure) as an important direction for future investigation.

[Q1 Q5] Additional Subject Baselines

Thank you for the valuable suggestions regarding regularization for more robust and fair biometric decoding baselines.

In response, we conducted additional experiments using MLP classifiers with varying levels of regularization and architectural complexity. As shown in the table below, we found that adding L2 regularization and ReLU activation significantly improves subject classification performance using raw fMRI voxel inputs. Notably, even a simple two-layer MLP with ReLU achieves near-perfect accuracy and MCC. We interpret this result as strong empirical support for our assumption that subject information in fMRI signals can be linearly disentangled from the joint latent space. We will elaborate on this point in our response to the linearity assumption.

[Q2] Linear Assumption Justification

We appreciate your concern regarding the justification of our linear entanglement hypothesis. Due to space constraints (10,000 characters), we were unable to include a detailed discussion in this response. We are truly sorry about this. However, Reviewer T9by raised the same point, and we have provided a thorough response there. We kindly refer you to our reply to T9by for further clarification on the detailed explanation and discussion of our linear entanglement hypothesis.

[Q4 Q6] Subspace Analysis and Robust Transformation

We thank you for raising the insightful questions regarding the interpretation of the object-specific latent dimensionality $d_\text{obj}$ and its effect on the model’s representation and dynamics.

Interpretation of $d_\text{obj}$ and Subspace Analysis

While $d\_\text{obj}$ is treated as a user-defined hyperparameter in our formulation, we conducted an additional analysis to investigate its representational efficiency using PCA on the object-specific features $\mathbf{Z}\_\text{obj}$ with $d_\text{obj}=700$. We computed the cumulative variance explained and found that:

• The first 10 components explain approximately 92.27% of the variance;
• 30 components capture over 98.05%, and
• Over 99.9% of the total variance is explained by fewer than 120 dimensions.

Along with the performance stability shown in our ablation in Appendix A.6, this suggests that although we allocate 700 dimensions to $\mathbf{Z}\_\text{obj}$, the intrinsic dimensionality of the object-specific subspace is *significantly lower. The fact that a small number of principal components dominate variance implies that the object subspace learned by our model is highly structured, low-rank, and compressible, indicating meaningful latent representations rather than arbitrary overparameterization. The model itself is not sensitive to the exact choice of $d_\text{obj}$ beyond a minimum threshold. These findings open up exciting potential for further exploration. In future work, we plan to study the semantic interpretability of leading PCA axes, examine temporal and dynamic properties of the object representations, and probe whether certain principal components correspond to high-level object categories or visual features.

Robustness of Basis Transformation

Regarding line 352, we agree that the term robust should be more precisely clarified. This robustness suggests that the disentangled space effectively captures semantic content across a range of dimensions in terms of dimensionality. We will revise the wording to effective or consistent if you feel robust is over-claimed here.

[Q3] Abstract Revision

Thanks for pointing this out. We will revise our abstract to avoid ambiguity and include the following contents accordingly: While reconstruction tasks leverage powerful generative models to produce high-fidelity images from neural recordings, they often pay limited attention to the underlying neural representations and rely heavily on pretrained priors. As a result, they provide limited insight into which neural components encode semantic content or how these representations vary across individuals. Our work shifts the focus toward understanding the structure and variability of semantic encoding in the brain, rather than merely replicating visual appearance.

In addition, we deeply appreciate your recognition of our work's potential to offer deeper insights into visual functionalities. To further strengthen the paper, we would be greatly thankful for a brief clarification on which specific aspects of neural visual processing could be more thoroughly illuminated. Your expert suggestions would be invaluable in guiding us to enhance the biological interpretability of our findings, whether in this revision or future work, while maintaining our core focus on discriminative decoding.

We sincerely appreciate Reviewer T9by for your valuable feedback. Below are our responses. (W and Q stand for Weakness and Question, respectively.)

[W1] Linear Entanglement Assumption

We fully acknowledge that the brain’s functional dynamics are fundamentally non-linear and complex. Our assumption—that subject-specific and object-specific components are linearly entangled at the latent feature level—is a simplifying inductive bias introduced to enable interpretable, computationally tractable disentanglement. We *do not claim it fully captures the richness of brain representations; rather, it serves as a first-order approximation that enables clear factorization of subject identity and semantic content from fMRI signals.

Empirical Justification

While we agree this assumption may not fully capture the brain's non-linear functional structure, we provide the following empirical evidence to support its pragmatic utility in our framework:

• Near-perfect biometric decoding (Table 2 in main paper): We achieve 99.9% accuracy in subject classification, indicating subject-specific features can be linearly separated from entangled fMRI features in the context of multi-subject learning;
• Subject-invariant object activations (Fig. 1 in main paper): After disentanglement, we observe that object activation patterns in the latent space exhibit minimal cross-subject variability, indicating that our method successfully distills stable, shared semantic representations—further supporting the effectiveness of linear disentanglement.
• Shallow MLP baselines confirm separability (Table below): We additionally implemented two-layer MLPs with ReLU activation for subject classification from raw fMRI. Surprisingly, even this lightweight non-linear model achieved near-perfect accuracy, validating the notion that subject-specific features are already largely linearly separable at the voxel level. This further justifies our design, especially considering our ViT encoder has significantly higher representational capacity compared with MLPs.

Discussion of Limitations

We agree it is important to discuss potential implications if the linearity assumption does not fully hold:

• If subject-object interactions are fundamentally non-linear, the disentangled object representation $\mathbf{Z}_\text{obj}$ may still retain residual subject-specific information, potentially introducing subject bias in semantic decoding and *diminishing our model's generalizability for unseen subjects.
• Conversely, enforcing strict linear disentanglement may suppress relevant non-linear object features in $\mathbf{Z}_\text{obj}$, potentially smoothing out sharp voxel-object modulations or degrading decoding performance for fine-grained categories.

Future Directions

We are actively exploring non-linear disentanglement techniques—e.g., manifold learning, kernel-based projection, or contrastive objectives—to model richer subject-object interactions while preserving interpretability. Incorporating auxiliary data such as structural MRI or resting-state connectivity could also regularize the subject basis more effectively. As suggested, we will make these limitations and future directions more explicit in the revised manuscript, particularly in the Discussion and Limitations sections.

[W2] Role of CLIP and fMRI

We thank you for this thoughtful suggestion. Further clarifying the role of the fMRI signal in the cross-attention fusion is essential for us to highlight our contribution.

To answer directly: in our fusion framework, the fMRI signal plays the role of "what to attend to" rather than simply "where to look." The fused output remains fundamentally neural in nature, modulated by subject-specific brain responses rather than being dominated by CLIP features. This is reflected in the design of the cross-attention module (Eq. 9), where CLIP embeddings serve as queries, while fMRI features act as keys and values—allowing neural responses to govern which semantic components are emphasized. This design achieves two complementary goals:

• Semantic Prioritization: CLIP features encode subject-invariant, stimulus-driven semantic content and contain rich spatial and conceptual structure. During cross-attention, CLIP features query the fMRI signal, and the fMRI effectively responds with what concepts or object components are prioritized, based on subject-specific neural responses. In other words, attention weights reflect which parts of the CLIP space align with neural activations.
• Subject-Specific Modulation: As shown in Fig. 4 (columns 3–10), residual attention maps reveal that fMRI signals modulate the CLIP-based predictions in a subject-dependent manner. This synergy allows our model to preserve semantic content from CLIP while capturing individual-specific neural patterns, highlighting how different subjects may focus on different semantic attributes of the same visual input.

Thus, rather than acting as a passive filter or spatial mask, the fMRI features dynamically shape which aspects of the CLIP semantic space are elevated—creating a bi-directional synergy. CLIP provides a semantically grounded reference frame, while fMRI reveals how the brain selectively engages with that semantic structure.

[Q2] Comparison against MindBridge

We thank you for this excellent suggestion and fully agree that including additional adapted baselines could strengthen the evaluation. In response, we adapted MindBridge—a reconstruction-oriented model—into a semantic decoding baseline by replacing its generative output head with a linear multi-label classification head, trained on its latent representations. While MindBridge was not originally designed for classification, this setup enables a fairer comparison of its representational utility under our semantic decoding task.

Due to time and resource constraints, we followed MindBridge’s original training protocol and evaluated the model on 4 subjects (1, 2, 5, 7). In contrast, our model (iMIND) was trained and evaluated on all 8 subjects, using the same data split and evaluation metrics. The results are shown below:

Despite the limited setup, these results demonstrate a substantial performance gap, especially in the fused modality setting. This gap reinforces a central argument of our paper: semantic decoding—particularly in a multi-subject setting—requires models explicitly designed to handle subject variability and semantic disentanglement. Simply reusing a reconstruction model, even with strong latent priors, is insufficient.

We will include this analysis and baseline comparison in the final version to strengthen the experimental evaluation and better contextualize iMIND’s advantages.

Thank you again for your time and valuable feedback on our paper. We wanted to kindly check if you have had a chance to review our rebuttal responses or if there are any points we could clarify further to assist your evaluation. We would be happy to continue the discussion.

We thank Reviewer 7Y8L for your constructive feedback. Below are our responses. (W, Q, and L stand for Weakness, Question, and Limitation, respectively.)

[W1] Motivation Rephrase

Thanks very much for your great suggestion. We admit that the original introduction placed too much emphasis on downstream applications such as autonomous driving and medical diagnosis, which may distract from the core neuroscience motivations of our work. In the final version, we will rephrase the introduction to better highlight key neuroscience-driven tasks—such as how the human brain encodes, disentangles, and generalizes semantic representations across individuals—and to emphasize the role of fMRI as a powerful non-invasive tool for probing the visual processing pipeline and subject-specific perception mechanisms. We will highlight our goal of contributing to a deeper understanding of multi-subject neural decoding and inter-individual variability in visual perception. We will retain only one or two sentences (in the conclusion) about potential long-term implications for human-AI interaction, as suggested by recent NeurIPS trends in brain-inspired AI.

[W3 Q1 L2] Empirical Validation of Disentanglement

This is a great point to make our work complete. To rigorously validate disentanglement, we have included the following results:

• Near-Zero Linear Correlation: We calculate the correlation of each feature vector of $\mathbf{Z}\_\text{obj}$ and $\mathbf{Z}_\text{subj}$ and report their mean and standard as $0.0027\pm0.0012$, supporting our disentanglement claim;
• Orthogonality Enforcement: The Frobenius norm $||\mathbf{B}\_\text{obj}\mathbf{B}\_\text{subj}^\top||= 2.7997\times 10^{-5}$ along with our orthonormal loss $\mathcal{L}_\text{orth}=7.2874\times10^{-5}$ confirms the role of learned $\mathbf{B}$ as an orthonormal basis for object-subject disentanglement.

[W2 Q3] Baseline Comparisons

We have expanded our comparisons to include state-of-the-art self-supervised fMRI encoders and strengthened our biometric decoding baselines. Below, we summarize the key results:

• For semantic decoding and neuro encoding methods, we evaluated three more self-supervised fMRI encoders, all trained with identical data/hyperparameters as shown below. We found that the Masked Autoencoder (MAE), which serves as the backbone of our model, outperforms the other three self-supervised methods. This is primarily due to its ability to reduce noise and redundancy inherent in fMRI data, as also observed in Mind-Vis. Furthermore, since prior work on semantic decoding—particularly for both classification and multi-subject modeling—is limited, we include MindBridge for comparison, even though it was originally designed for reconstruction tasks. To adapt MindBridge for classification, we extract its output features and train a linear classification head on top. This comparison confirms the need for task-specific architectures like ours that are explicitly designed for semantic decoding across subjects.

• For biometric decoding, we included three additional baseline models. Notably, we found that even a simple two-layer MLP with ReLU activation and L2 regulation can achieve perfect subject classification from fMRI voxel signals. This result provides strong empirical support for the core assumption of iMIND—that subject information in fMRI can be linearly separable at the latent feature level. Given that the Vision Transformer (ViT) offers significantly greater feature extraction capacity than a shallow MLP, this finding reinforces the validity of our disentanglement approach.

[W5 L1] Dataset Generalization

We appreciate your comment on the generalization of the data set. We agree that demonstrating cross-dataset generalizability is an important step toward establishing a more comprehensive and robust framework.

In exploring additional datasets, we considered both the BOLD5000 and HCP movie-watching datasets, which involve naturalistic visual stimuli and multiple subjects. However, two key challenges currently limit their use in our study:

• Lack of semantic decoding benchmarks (BOLD5000): Unlike NSD, which provides well-curated annotations and is widely used for semantic decoding tasks, BOLD5000 lacks standardized, reproducible benchmarks for object-level classification. This makes it difficult to evaluate and compare performance under our semantic decoding setup.
• Computational challenges (HCP): The HCP movie dataset includes 158 subjects and more than 800 semantic concepts. Given the need to model multi-subject disentanglement and high-dimensional voxel features, applying our method to this dataset is computationally demanding. Unfortunately, due to limited time and resources during the review period, we were unable to complete a full evaluation.

That means, we recognize the importance of cross-dataset validation and plan to explore generalization on datasets like HCP and BOLD5000 in future work. We will explicitly acknowledge this limitation in the revised manuscript and elaborate on it in the Limitations and Future Work section.

Thank you for taking the time to review our rebuttal. If there are any points where we could further improve clarity or address remaining questions, please don’t hesitate to let us know. We would be happy to continue the discussion. Thank you again for your thoughtful feedback.