MindDETR: Beyond Semantics, Exploring Positional Cues from Brain Activity

W1: For brain visual decoding, pixel-level reconstruction/decoding is currently very challenging. We hope to provide some insights through the proxy task of determining object location/boundaries. In neuroscience, the significance of decoding tasks is quite limited, including previous reconstruction methods, which are more of an attempt and verification. There is also relatively little work on the encoding and decoding of location information. We hope to inspire other researchers.

W2: Limited performance can be attributed to the NSD dataset’s lack of focus on object detection during its collection phase. This oversight means that many small and medium-sized objects in the visual stimuli may have been overlooked by participants, making it difficult to extract their location information from MRI data. Additionally, the dataset’s size is inadequate; it has been observed that a participant has seen no more than 10,000 images, which is typically insufficient for a model to develop robust generalization capabilities through training for more complex computer vision tasks. This limitation applies to our model as well, given its parameter count is comparable to that of DETR for visual object detection.

W3: See 'To Reviewer 5Sff - W5'.

Q1: To measure the consistency in semantics and location, we performed the following calculations:

For a fixed IoU threshold, we analyzed the detection results of two different subjects on the same visual stimulus image. If a bounding box from one subject and a bounding box from another subject share the same category and their IoU meets or exceeds the threshold, the two bounding boxes are considered a match. We then averaged the matching rates across all subject pairs to obtain the final consistency score, as illustrated in Fig.10 in A.8.

It can be seen that the detection stability of MindDETR is significantly higher than that of MindEye.

Q2: See W3.

We thank Reviewer T78L for his/her valuable feedback, and here we provide corresponding responses to address these concerns.

W1: We totally agree with these two comments. Given that the mental images of participants may significantly differ from what they perceive visually, achieving an exact reconstruction of the original image is inherently challenging. Consequently, we have introduced this proxy task, aiming to decode MRI signals from alternative perspectives. We also believe that the detection task can work with reconstruction task (as discussed in Section 5.2 limitation and future work). For example, by incorporating positional information as an additional condition, reconstruction accuracy could be further improved, as suggested in related works [1, 2].

References:

[1] Feng, Yutong, et al. "Ranni: Taming text-to-image diffusion for accurate instruction following." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

[2] Yun, Jooyeol, et al. "Generative Location Modeling for Spatially Aware Object Insertion." arXiv preprint arXiv:2410.13564. 2024.

W2, W3:

Here is the detailed explanation of line 310:

Firstly, we resize the reconstructed images from Takagi and MindEye to a standard size of 425x425 pixels, which matches the visual stimulus' size in the NSD dataset. Then we conduct object detection using DAB-DETR on these reconstructed images. Before detection, to align the dimensions of the input image, the images will be reshaped to 800 * 800 pixels. After detection, the detected bounding boxes will be used to calculate the metrics with the annotated bounding boxes of the original data stimulus images. The calculation rules for the metric are consistent with COCO.

We have incorporated two additional models, DINO[1] and CO-DINO[2], for detecting the reconstructed images, which exhibit better detection performance on small objects. The results are shown in Tab.4 in A.6.

As shown in the table, the choice of detection model has a noticeable impact on the results. However, a stronger detection model does not always yield better performance and may even cause slight metric regression. We hypothesize that this is because stronger detection models are more likely to identify blurred objects in the reconstructed images generated by MindEye, which weaker models might overlook. The inaccurate positional information associated with these blurred objects could contribute to the observed decline in metrics.

References:

[1] Zhang, Hao, et al. "DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detection." The Eleventh International Conference on Learning Representations.

[2] Zong, Zhuofan, Guanglu Song, and Yu Liu. "Detrs with collaborative hybrid assignments training." Proceedings of the IEEE/CVF international conference on computer vision. 2023.

W4: Thanks for this suggestion. To explore the role of visual regions (ROIs), we trained and tested using only the voxels from V1, V2, V3, and V4, respectively, yielding the results shown in the following table:

The number of voxels in V1, V2, V3, and V4 progressively decreases, with approximate ranges of 3000–4000, 2000–3000, 1500–2000, and 1000–1300, respectively. However, the detection results show an inverse trend, where higher visual areas (V3 and V4) outperform lower visual areas (V1 and V2). Additionally, the difference in the AR metric is smaller than that in the AP metric, as shown in the table. This indicates that the higher visual areas, V3 and V4, which undergo more advanced neural encoding, contain richer information and are easier to decode for object location and category. In contrast, while the lower visual areas, V1 and V2, also retain some information (as reflected by an acceptable AR metric), factors such as noise make decoding object location and category more challenging, resulting in a lower AP metric.

Thanks for your valuable feedback, and here we provide corresponding responses to address these concerns.

W1: Our motivation stems from addressing the challenge of inaccurate positional features in brain signal decoding during image reconstruction tasks. However, this is an exceedingly complex problem, and we aim to inspire and encourage the research community to engage with this promising research direction by introducing this proxy task.

First, as shown in the experimental results, the proposed proxy task optimizes the models based on the positional-aware objective, thus improving the accuracy of position decoding. Second, the brain detection results obtained through our proxy task have the potential to enhance positional awareness in reconstruction task. Current reconstruction methods generally use decoded CLIP embeddings as conditions for Stable Diffusion models. By incorporating positional information as an additional condition, reconstruction accuracy could be further improved, as suggested in related works [1, 2].

First, although the AP50 metric for this task is relatively low, this is primarily due to the presence of a large number of small objects in the NSD dataset (sourced from COCO), which reduces the average localization accuracy. These small objects are often overlooked in the primary evaluation metrics (semantic consistency) for image reconstruction tasks. Second, our preliminary exploration demonstrates that detection-based methods significantly outperform reconstruction-based methods in terms of localization accuracy, indicating that the quality of the existing dataset can support brain decoding with higher localization precision. Third, the current approach is a baseline model for future work, and we believe there is substantial room for improvement.

References:

[1] Feng, Yutong, et al. "Ranni: Taming text-to-image diffusion for accurate instruction following." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

[2] Yun, Jooyeol, et al. "Generative Location Modeling for Spatially Aware Object Insertion." arXiv preprint arXiv:2410.13564. 2024.

W2: During the collection of the NSD dataset, subjects were asked to observe and recall the details of the images as much as possible. At the same time, subjects were fixed during observation, so the positional information of the objects exists. Although the current results remain relatively low compared to image-based detectors, the proposed task and baseline model demonstrate improved fidelity in spatial information decoding over existing reconstruction methods. This indicates that the potential of spatial information decoding in the current NSD dataset has yet to be fully exploited. At the same time, we acknowledge that more systematic data collection strategies—such as requiring participants to pre-memorize location-specific features—could further advance progress in this field.

W3: The role of the convolution kernel in our model is similar to that in the visual CNN backbone. Generally, smaller convolution kernels (such as 3x3 or 5x5) are often used to capture local features, while larger convolution kernels (such as 7x7) may be used to capture broader contextual information. In MindDETR, we found that the generalization ability of the linear layer is insufficient, so we added a convolutional layer to extract local features. Determining the size of the convolution kernel through experimentation is a common practice in the design of visual models.

We also acknowledge that both object detection and cross-modal distillation are established techniques in deep learning. However, the contribution we would like to emphasize more is: prior image-reconstruction based methods have largely overlooked the evaluation of spatial information in their metrics, with a focus of the semantic consistency of the main object. This omission may lead researchers in the field to misjudge the fidelity of brain signal decoding. In this work, we build a proxy task and provide a baseline model to encourage the future works on this field.

W4: MindEye focuses on image reconstruction and does not directly provide object location information. To address this, we used a pre-trained object detection model to detect objects in the images reconstructed by MindEye, enabling us to extract object location data from the reconstructions.

Notably, during training, MindEye did not utilize bounding box information from the dataset. Its primary objective is to predict or reconstruct the CLIP visual embeddings of visual stimuli. The core of MindEye's training process relies on a contrastive loss, which emphasizes semantic consistency of embeddings. In contrast, our model is specifically designed to decode location information, enabling it to achieve better performance in this regard compared to MindEye.