Raptor: Scalable Train-Free Embeddings for 3D Medical Volumes Leveraging Pretrained 2D Foundation Models

We thank the reviewer for their comments. We also agree with the reviewer that our novel approach has several implications for the field, as it would make the computation of embeddings for volumes much faster while maintaining the ability to perform downstream tasks. We also appreciate the reviewer’s evaluation of the theoretical background of our approach.

Why DINO works for medical data:

The high-level intuition is that the semantically meaningful features of medical volumes are still part of the universal image features (edges, textures, shapes), which a broad 2D encoder has already learned from large‐scale, diverse data. RAPTOR uses generic image foundation models like DINO to capture any meaningful image features and performs low rank approximations to retain only the relevant ones. This is in contrast to many existing medical image/volume models that try to learn such mappings from scratch or fine-tune a general model to be biased towards them.

To illustrate, we projected both generic (ImageNet) and medical (MedMNIST) embeddings onto the principal components derived from the generic dataset. In the table below, we calculate the total variance explained as we add more PC’s. Although the medical data do not align with the very top principal directions, they catch up as more components are added (eventually matching the generic variance). This demonstrates that medical image features lie within the general “image” space.

Why we don’t use another method to aggregate slices:

We explored the simplest approach—summing and random projection—because it is computationally lightweight and maintains near‐orthogonal signals from the 2D tokens without additional training (also note that averaging/summing is identical in expectation to random projection into dimension 1). More structured dimension‐reduction techniques may introduce significantly higher computational or memory costs, especially for high‐resolution volumes. We agree that specialized subspace methods are an interesting direction for future work, particularly if they exploit known volumetric structures. As an example, please see our response to reviewer zwUQ on partitioned embedding.

Time taken to run Raptor:

We appreciate the reviewer’s interest in a more detailed efficiency comparison. In the table below, we provide approximate times for fine-tuning, inference, and one-time embedding extraction across Raptor, SuPreM, and Merlin when training on the Organ dataset (971 volumes). While Raptor does incur a one-time cost to produce embeddings, its subsequent fine-tuning is minimal (logistic regression or a small MLP) and can often run on CPU. This contrasts with end-to-end 3D methods (SuPreM, Merlin) which must either be entirely trained or extensively fine-tuned, typically on a GPU with longer runtimes.

Stability of random projections:

We also tested more seeds (up to 10) for the 3D MedMNIST dataset to assess the stability of random projection. In the table below, we observe consistently low standard deviations, suggesting that random projections remain highly reliable in practice:

We hope that the additional analysis provides better insight into the capabilities of our method. If these clarifications have addressed any lingering concerns, we would sincerely appreciate if the reviewer would consider raising their score.

We appreciate the reviewer's keen observations and suggestions. Thanks to their recommendations, we were able to further uncover Raptor's capability in a challenging task (segmentation) and are excited for future directions.

statistical tests (e.g. p-values) … especially for small sample settings

We acknowledge the value of statistical tests for clarifying performance differences and intend to include in a future revision. Given the strong performance of existing methods, we did not observe any significant improvement over the next-best method under the bonferroni-corrected threshold (p < 0.05/19, reflecting 19 total datasets). However, we did observe some modestly significant improvements (p < 0.05). Among the regression datasets, we observed such improvements in four of the datasets (4/10). We did not observe any such improvements for the classification datasets.

In terms of the subset experiments (varying training set size from 10~500), we similarly found modestly significant improvements (p < 0.05) in 3/5 settings for the UKBB white matter dataset, 1/5 in CCCC-II, and 2/5 in Synapse.

Is there any bias per digit in the simulations?

We appreciate the suggestion to control for digit shape in our simulations. Accordingly, we fixed the digit to either 0, 1, or 8 and generated simulations for each case for benchmark. We observed that the choice of digit can slightly influence the results. Here we demonstrate these results with 300 generated training samples each, as the simulation and evaluation process is time consuming end-to-end.

Segmentation

We concur with the reviewer’s point that more downstream tasks need to be attempted. We have now conducted additional experiments on the Medical Segmentation Decathlon (https://www.nature.com/articles/s41467-022-30695-9), focusing on four tasks: hippocampus, spleen, colon, and hepatic vessel segmentation. Each task presents unique challenges (e.g., limited contrast for hippocampus, class imbalance for hepatic vessels). For Raptor, we trained a 2-layer convolutional head to learn a transformation of the embeddings to the segmentation.

As shown here, Raptor achieves competitive results compared to MedSAM (a dedicated segmentation model). Notably, on the Hippocampus dataset, Raptor surpasses MedSAM’s Dice (0.719 vs. 0.615) and IoU (0.607 vs. 0.575), and performs similarly for hepatic vessels. MedSAM, however, excels on spleen and colon, which have fewer training samples. Although our primary goal was to obtain general-purpose embeddings, these early results suggest Raptor can also serve as a reasonable foundation for volumetric segmentation, without requiring large‐scale 3D training.

Figure 1 clarity

We agree Figure 1 could be improved in clarity and will make the requested adjustment given the opportunity.

Would orthogonal feature processing compromise inter-slice contextual features

We appreciate this important point raised by the reviewer. We provide an intuition how the resulting Raptor embedding can still retain volumetric signals. For a raptor embedding $e$, a tensor of dimensions $3\times100\times16\times16$, the coordinate $(x,y,z)$ in the volume can be represented by three slices—one per axis—yielding tokens indexed by, e.g., $e[0,:,x,y]$, $e[1,:,y,z]$, and $e[2,:,x,z]$. Each patch within these slices is mapped into the K-dimensional embedding space; subsequent MLP layers can then fuse these partial 2D perspectives into a more coherent 3D representation. Our quantitative results on datasets like UKBB suggest that Raptor’s multi-view scheme does effectively capture enough high-level volumetric information for robust performance.

Clearer workflow description on using Raptor embeddings

This is an important suggestion and we plan on including a diagram and a detailed description of the Raptor pipeline for downstream tasks.

We thank the reviewer for the insightful questions, and hope our response addresses any remaining concerns. If so, we would greatly appreciate it if the reviewer could reflect these clarifications by raising their score of our work.

We thank the reviewer for their time and constructive comments, as well as their positive assessment of our method’s clarity and novelty. As noted by the reviewer, Raptor introduces a paradigm that demonstrates many advantages beyond existing works, and we verify our claims with a wide range of empirical results.

Average score over 10 brain regions:

For better clarity, we have now added a column for average performance on the regression tasks, reproduced below. Overall, Raptor most accurately predicts physiological measures given brain MRIs, and Raptor-B comes in as the second best (best score in bold, second best italicized).

We again appreciate the reviewer for their words of support for our work. In addition to the improved table, we hope that the additional analyses we shared with the other reviewers have further demonstrated the capabilities of our method, and if deemed so, humbly request that the reviewer consider raising the score as a further vote of confidence.

(Below is in response to a point raised by reviewer zwUQ; we have moved this here due to the space limit, but we think that reviewer 28uY would also be interested to know.)

More evidence for effectiveness with limited training samples:

We agree that a single dataset is insufficient to support our point. Hence we conducted the same experiments (varying training set size) for the CCCC-II dataset (classification) and the white matter category of the UKBB dataset (regression). As shown in the table below, while some methods eventually catch up as the sample size grows, Raptor maintains a clear advantage in low-data regimes for both tasks, underscoring our original claim of data efficiency (best score in bold, second best italicized).

UKBB White Matter ($r^2$)

CCCC-II (AUROC)

We thank the reviewer for their detailed assessment of our manuscript. We appreciate that they viewed our experiments as supporting the utility of our method, and we agree that we could support some of our claims with additional experiments. We address each of the points raised below.

Meaning of scalable in our work:

We use the term “scalable” to highlight some key aspects of Raptor:

1. It can process high-resolution 3D medical volumes with only a single pass through a fixed 2D encoder, avoiding the heavy computational cost of purely 3D models.
2. The framework can readily scale to larger or more diverse medical datasets and reuse the embeddings for multiple downstream tasks, as there is no training involved.
3. The inference step can be easily parallelized since we only require a frozen foundational model and random projections.

More evidence for effectiveness with limited training samples:

Due to the word limit, we have moved the subset results to our response to reviewer 28uY.

Why not sample from N(0, 1/k):

In terms of choice of noise for the random projections, we proceeded with the numpy default which was N(0, 1). Quantitatively we don’t expect a notable difference choosing a variance in this range, as the embeddings are eventually standardized in the fine-tuning step.

Exploration of medical VLMs:

While Llava-Med and Med-Flamingo are strong multimodal methods in their respective domains, their encoders process 2D data, making direct comparison on volumetric (3D) tasks ambiguous (unlike Merlin, which was designed for 3D). Nonetheless, we tried substituting their image encoders into our pipeline in place of DINO: Raptor-LVM and Raptor-CLIP (note that Med-Flamingo leverages OpenAI’s CLIP as the image encoder, which we have previously benchmarked as an alternative to DINO).

(AUROC shown)

These alternatives show competitive performance but do not surpass DINO (our current proposed approach).

Parameter number:

We would be happy to include this information in our final version.

Plot upper bound in scaling figures:

We appreciate this suggestion, and plan to revise our figures accordingly.

Can accuracy be improved by removing problematic views:

We appreciate this interesting suggestion. We explored whether skipping a view might improve FractureMNIST performance (either AC, CS, or AS, instead of ACS). However, results show no improvement: AC (0.657), CS (0.654), and AS (0.660), all perform below the default ACS (0.677). Intuitively, removing an entire view discards potentially useful information, even if that view is somewhat degenerate.

An alternative solution would be to partition slices within a problematic view. For instance, instead of aggregating all 64 slices at once, we could split them into two groups of 32 slices each and perform Raptor, then concatenate. Although this could mitigate partial cancellation effects, it increases embedding size. We find this to be a promising extension to Raptor in both practical deployment (handle slice-wise misalignment) and theoretical analysis (quantify error bounds under partitioning). We are actively exploring how best to balance these tradeoffs.

Why MAE performs bad:

The choice of architecture for the MAE was a transformer with 3-dimensional positional encodings. Despite our efforts to tune the model, we suspect that the datasets are simply too small (to learn e.g. effective patch embeddings and relationships between positional encodings).

VQA Captioning:

Based on the strong performances that we observed (+ segmentation, in response to reviewer BpWD), we do hypothesize that Raptor embeddings would be capable of captioning. In this work, we focused on thoroughly verifying several properties of Raptor, and hope to evaluate its novel use cases in the future.

We appreciate the insightful questions raised by the reviewer, and hope that the additional information we have provided further supports the potential of our method. If our response has sufficiently resolved current questions regarding our work, we would appreciate an increase in our score.