M3D: Advancing 3D Medical Image Analysis with Multi-Modal Large Language Models

• For the training of their proposed model, the authors used the Radiopedia dataset. However, radiopedia explicitly prohibits scraping the data from their website for such uses. https://radiopaedia.org/terms . It is mentioned that "Radiopaedia is not designed or intended to be used as an imaging data-set for Machine Learning/Artificial Intelligence research, nor is it available for bulk download using an automated script or robot or any other process whereby many cases are downloaded." Copyrights to data is clearly a very active topic currently, and it is problematic that the model developed here does not abide by the original copyrights of a crowd-compiled dataset. Overall, it does not seem that the authors have the ability to download and freely distribute the dataset as claimed.
• No comparisons are performed with more relevant task specific models or more stat of the art 3D vision language models. From a baselines perspective, it would be beneficial to compare to the BiomedCLIP model for retrieval as well as training. For segmentation, using more common models like nnunet or swinunetr would be more fair. For general 3D VL training, there is no comparison to a couple of recent papers that are very related: 1. Hamamci et al. Developing Generalist Foundation Models from a Multimodal Dataset for 3D Computed Tomography. And 2. Blankemeier et al. Merlin: A Vision Language Foundation Model for 3D Computed Tomography
• The M3D-RefSeg is the TotalSegmentator dataset. All that was done as a mapping between class number and the text description. Rebranding it as a unique contribution is not appropriate.
• Table 1 describes a Texts column - it is unclear what this is. Tokens? Also, referring to segmentation masks seems unlikely to be considered as text since its a specific class anyway at the end of the day.
• CLIP struggles with long context and fine grained learning. It does not seem that there was any additional method used besides the traditional CLIP in this study, which may be problematic because of the high density of text information and low variability of the images.
• It is unclear to me what the value of the benchmarking positioning task is, when one could simple perform the segmentation for the same tissues anyway? Since the bounding box is generated from the segmentation label, there is limited new value provided.

Lack of Explanation for Resizing Method: The authors mention using Min-Max normalization and resizing 3D CT images to a standard dimension of 32×256×256, but the specific resizing technique used (e.g., interpolation method, aspect ratio considerations) is not clarified. Without this information, it is difficult to assess how the resizing might affect spatial integrity in 3D images, especially for subtle structures critical in medical analysis. No Comparative Study on Resizing: It would be beneficial if the authors conducted a comparative study to evaluate the effects of different resizing methods on performance, as variations in preprocessing can impact model accuracy on medical data.

Lack of Comparative Analysis with MLP: The paper introduces a 3D Perceiver module to handle token reduction for efficient 3D processing, but it does not compare this approach directly against a simpler alternative, such as a Multi-Layer Perceptron (MLP). A controlled comparison would demonstrate the added value of the 3D Perceiver and validate its effectiveness in preserving spatial information while reducing computation. No Ablation Study on the 3D Perceiver: Including an ablation study that isolates the impact of the 3D Perceiver module would strengthen the paper by empirically verifying its contribution to the model’s performance.

Limited Comparative Analysis with 2D Models: Although the paper focuses on 3D models, a more detailed comparison with state-of-the-art 2D models on similar tasks (e.g., segmentation or retrieval) would highlight the advantages and limitations of the 3D approach. By doing so, the authors could clarify whether the shift to 3D improves performance in specific areas, such as spatial recognition, and where it might face challenges, such as increased computational costs.

Lack of Discussion on Demographic and Disease Variety: While M3D-Data is substantial in size, the paper does not address whether it includes diverse demographic groups or a wide range of diseases, which are crucial for generalizable models in healthcare. An analysis of the dataset’s coverage across demographic and pathological categories would provide insight into its representativeness and the potential for biases. Absence of Evaluation for Dataset Representativeness: Ensuring that M3D-Data includes representative samples across age, gender, and disease types would enhance the model’s applicability. Without this, there’s a risk that M3D-LaMed may perform less effectively on underrepresented groups, limiting its utility in real-world clinical applications.

• Computational Efficiency: handling 3D images and text is known to be computationally demanding. Authors could have developed more on the problems that could arise if using longer text sequences or higher resolution images. • How well M3D-LaMed works on other medical imaging modalities? • The architecture of M3D-LaMed includes a 3D image encoder, a perceiver, and LLM but the interaction and importance of those three components is not fully explained.

The weakness of this paper can be concluded as follows:

1. As the model is focused on using 3D image analysis, there is a lot of 3D foundation backbone proposed such as 3D swin transformer, 3D large kernel convolution, there is limited discussion in this paper on why it is important to use ViT here.
2. As the paper benchmarks the performance across different models, only a few baselines comparisons is performed with the adjustment on LLMs and one model with respect to the corresponding task.