Orochi: Versatile Biomedical Image Processor

First and foremost, we would like to thank the reviewer for recognizing the substantial effort invested in our work and for the careful and diligent review. We do include a Limitations section in our manuscript, and here we will provide point-by-point responses to all comments. We aim to elaborate on aspects that were condensed due to page limitations and hope to earn your further approval by addressing your questions. Thank you!

Q1:

Biomedical image modalities can be quite different. The reader is not clear why features learned on one modality (e.g., microscopy cell images) can be used for the other modality with a large domain difference (e.g., MRI on brain images).

A1: As we articulated in the first paragraph of our introduction, the primary processing challenges for biomedical images, compared to natural images, stem from two main aspects. First is the issue of imaging device operational trade-offs: long imaging times can damage the subject, while shorter times may compromise image quality. This trade-off is a universal challenge present in both microscopy and clinical imaging. The second is the shortcomings of imaging modalities. For instance, MRI is superior for soft tissue imaging, while CT excels at imaging hard structures; many such complementary modalities exist.

Therefore, different imaging modalities are not entirely disconnected. They share a degree of commonality and complementarity, from their underlying imaging principles to their imaging content. This understanding motivated our decision to broadly increase the diversity of our training data.

Furthermore, our experiments demonstrate that a model pre-trained in this manner exhibits superior efficiency and performance when fine-tuned on various downstream tasks. This not only validates our hypothesis but also aligns with the broad consensus in the general vision domain that pre-training on diverse datasets like ImageNet leads to better performance for foundation models.

Q2:

Some biomedical image tasks require paired images of different modalities on the same object. As the collected 100+ datasets are quite heterogeneous, how can this problem be solved?

A2: We understand your concern. While downstream tasks like registration and fusion often require paired images, collecting such data at a large scale for pre-training is challenging. To address this, we developed targeted self-supervised strategies:

• For Registration: We create a proxy task by applying multi-scale non-rigid deformations to an original image. The model's pre-training objective is to generate the registration grid that reverses this deformation. When fine-tuning on a downstream task, which may have only a small number of paired samples, our model can leverage this pre-learned ability to generate registration grids. This helps it adapt faster and more effectively, mitigating the risk of overfitting that models without such pre-training would face.
• For Fusion: We build upon existing work that uses Masked Image Modeling (MIM) for fusion pre-training [1]. We advance this by employing a dual-masking strategy. This forces the model to first learn to fuse two non-overlapping masked regions of an image before predicting the content of the remaining masked area. Our ablation studies confirm that this method yields superior results compared to a standard MIM pre-training approach.

In summary, our pre-training phase does not require strictly paired data; instead, it learns through purely self-supervised methods. This pre-training paradigm allows us to scale up our dataset size and simplifies the process of incorporating new, unpaired data in the future.

Q3:

The method section is focused on the engineering implementation details. There is no new technique introduced. (Weakness # 3)

A lot of engineering details in the current method section can be moved to the supplement. (Question # 2)

A3: While we did describe our self-supervised strategies in detail in the Method Section, we respectfully disagree with the assessment that "There is no new technique introduced."

Our writing style was intentionally modeled after widely recognized papers that introduced influential self-supervised strategies, such as MAE [2] (whose Method Section details its mask ratio, mask token) and I-JEPA [3] (which follows a similar pattern). Our paper, in fact, features a more complex design of self-supervised strategies, and detailed descriptions are unavoidable.

Specifically, our novelty lies in several areas:

• Our Task-related Joint-embedding Pre-training (TJP) is a novel approach that simultaneously utilizes specific degradation types across multiple biomedical, domain-specific tasks as self-supervised learning strategies. To the best of our knowledge, this concept has not been previously proposed in existing literature.
• Within each self-supervised strategy, we introduced new considerations. For example, we extended single-mask image modeling [1] to a dual-masking approach to better suit fusion tasks. We also improved upon the single-scale image degradation scheme in DeepLP [4] by implementing a multi-scale version.
• Our Methods section also covers the key concept of our designs for data acquisition, model architecture, and post-training strategies. While we are not the first to use a Swin-Mamba framework, we are the first to apply it to this breadth of multi-task pre-training. Furthermore, our three-tiered fine-tuning strategy and its validation for different scenarios are also contributions. We will revise the manuscript to highlight these innovations more prominently in the main text in a future version with a more generous page limit.

While these designs certainly build upon the work of others, we believe the claim of "no new technique" is not a fair characterization, especially when compared to other papers that focus on advancements in self-supervised strategies.

Q4:

A long list of pre-training techniques are available in the community. Four pre-training techniques are assembled in this draft. The underlying motivations to choose these four can be discussed with more details and analyzed with more experiments.

A4: We chose these four pre-training techniques (for restoration, super-resolution, registration, and fusion) not only because they represent the four primary low-level processing tasks in biomedical imaging, but more importantly, because of their deep underlying relationships, which we took considerable space to explain.

The need to preprocess biomedical images stems from two fundamental inadequacies of current imaging methods:

1. Shortcomings of imaging modalities: A single modality often has limitations. Therefore, a model should learn to fuse information from different modalities, even if they are not perfectly aligned (which requires registration).
2. Imaging device operational trade-offs: Even with an adequate modality, we cannot guarantee high image quality. Thus, we want the model to extract useful information even from low signal-to-noise ratio images and perform image reconstruction (restoration and super-resolution).

In our ablation studies, we demonstrated that if our TJP is removed and replaced with standard MIM methods, the performance suffers, especially on spatially-aware tasks like registration. We have additionally performed new ablations comparing TJP to DINOv2 [5] pre-training. As shown in the results below, the TJP approach remains more robust.

Q5:

There are 100+ datasets with various tasks. When training the model, are all of them used at once for multi-task learning? Or, are a subset of them randomly sampled in each epoch? Or, is each of them iterated during the training? Some details on the training strategies can be added.

A5: Thank you for the suggestion. Yes, all of the studies were included for multi-task learning, but not all the raw data was used. It is important to clarify that the original studies associated with those data did not necessarily involve deep learning, meaning properties like data dimensionality and resolution may not be directly suitable for training a deep learning model.

To leverage this diverse data, our solution involved several steps. After an initial filtering phase (to remove non-imaging data like tables and graphs), we performed multi-scale random sampling on the source data (as detailed in Section 3.2 of our paper). We then employed a hybrid training strategy using both locally stored (fixed patch/volume) and streamed data (random patch/volume).

Reference

[1] Li, Jiayang, et al. "MaeFuse: Transferring omni features with pretrained masked autoencoders for infrared and visible image fusion via guided training." IEEE Transactions on Image Processing (2025).

[2] He, Kaiming, et al. "Masked autoencoders are scalable vision learners." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022.

[3] Assran, Mahmoud, et al. "Self-supervised learning from images with a joint-embedding predictive architecture." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.

[4] Fang, Linjing, et al. "Deep learning-based point-scanning super-resolution imaging." Nature methods 18.4 (2021): 406-416.

[5] Oquab, Maxime, et al. "Dinov2: Learning robust visual features without supervision." arXiv preprint arXiv:2304.07193 (2023).

We sincerely thank the reviewer for the recognition of the motivation, data selection, model training, and experimental presentation of our work. In the following, we provide point-by-point responses to all comments, elaborating on areas that we were unable to detail due to space constraints. We hope this addresses your questions and earns your further approval. Thank you again!

Q1:

Limited detailed information on the diversity and balance of the pre-training dataset: While the paper highlights the large scale (100+ public studies, 100TB+ of data), it lacks clarity on the dataset’s composition—such as imaging modalities, species, pathological vs. normal cases, resolution range, and artifact types. This makes it difficult to assess how broadly applicable the claimed versatility of Orochi truly is.

A1: We invite you to review the Data_Sources.xlsx file located in our Supplementary Materials, within the "dataset" folder. This file contains the organized metadata that covers most of the information you mentioned (modality, species, dimension, resolution, directory...). Our data diversity is primarily derived from the modality diversity of the Image Data Resource (IDR) [1] website (which includes Light-sheet fluorescence microscopy, Digital pathology imaging, Yeast studies, High-content screening, etc.) and is supplemented by Atlas datasets (such as the Human Organ Atlas [2] and the Human Induced Pluripotent Stem Cell Atlas [3]). Due to page limitations, the main body of our paper focused on task versatility, with many details on data versatility expanded upon in the appendix. We hope this clarification resolves your concerns.

Q2:

Failure cases are not discussed, making it difficult to understand the model’s limitations in real-world scenarios.

A2: Like all scientific papers, our work has its limitations, and we have already included a "Limitations" section in the manuscript. The nature of our training methodology and data types means that our model is primarily focused on low-level tasks. We have since conducted additional experiments by fine-tuning the model on two high-level tasks, namely segmentation and classification.

First, we conducted new tests on the model's performance on a subset of BTCV [4], REFUGE [5], and CadVidSet [6] segmentation datasets. We strictly adhered to the training hyperparameter settings of MedSAM2 [7] and carried out a complete fine-tuning following its GitHub repository. Baseline results are from the original paper.

Also, we conducted classification tasks on TissueMNIST (2D) and OrganMNIST (3D) datasets. All experimental setups are from the MedMNIST series papers [8] [9] [10].

Dice Score

We have demonstrated Orochi's capability to adapt to high-level tasks; however, we intend to avoid exaggerating its scope of application. We expect that its performance may decline to average or insufficient levels when applied to a broader range of high-level tasks, as our primary objective remains centred on the development of a unified low-level processor, as originally stated since the abstract of this paper.

Reference

[1] Williams, Eleanor, et al. "Image Data Resource: a bioimage data integration and publication platform." Nature methods 14.8 (2017): 775-781.

[2] Walsh, Claire L., et al. "Imaging intact human organs with local resolution of cellular structures using hierarchical phase-contrast tomography." Nature methods 18.12 (2021): 1532-1541.

[3] Viana, Matheus P., et al. "Integrated intracellular organization and its variations in human iPS cells." Nature 613.7943 (2023): 345-354.

[4] Landman BA, Xu Z, Igelsias JE, Styner M, Langerak TR, and Klein A, "MICCAI multi-atlas labeling beyond the cranial vault workshop and challenge," (2015)

[5] Orlando, José Ignacio, et al. "Refuge challenge: A unified framework for evaluating automated methods for glaucoma assessment from fundus photographs." Medical image analysis 59 (2020): 101570.

[6] Wang, Lu, et al. "Coronary artery segmentation in angiographic videos utilizing spatial-temporal information." BMC medical imaging 20.1 (2020): 110.

[7] Zhu, Jiayuan, et al. "Medical sam 2: Segment medical images as video via segment anything model 2." arXiv preprint arXiv:2408.00874 (2024).

[8] Yang, Jiancheng, et al. "Medmnist v2-a large-scale lightweight benchmark for 2d and 3d biomedical image classification." Scientific Data 10.1 (2023): 41.

[9] Yang, Jiancheng, Rui Shi, and Bingbing Ni. "Medmnist classification decathlon: A lightweight automl benchmark for medical image analysis." 2021 IEEE 18th International Symposium on Biomedical Imaging (ISBI). IEEE, 2021.

[10] Doerrich, Sebastian, et al. "Rethinking model prototyping through the MedMNIST+ dataset collection." Scientific reports 15.1 (2025): 7669.

Thank you for your detailed response. Most of my concerns have been resolved. However, I still have one remaining question. Could you provide more details on the failure cases in low-level tasks beyond what is currently discussed in the Limitations section? The section as it stands does not appear to cover specific failure examples. Since it may be difficult to present them qualitatively, describing representative situations or providing quantitative evidence would also be helpful.

Thank you for your response. I believe Orochi is a valuable contribution, especially in its focus on generality and multi-task capability. All of my concerns have been resolved, and I will maintain my current score.

We thank the reviewer for recognizing that our motivation is reasonable, the paper is well-written and easy to follow, and that our training methodology, data usage, and workload are appropriate for the topic. Despite the divergence in the final score, we would like to provide a detailed point-by-point response to address your concerns and hopefully resolve any outstanding questions.

Q1:

Actually, the practical value of low-level tasks is relatively low, especially when comparing to high-level tasks. Though there are commonly used metrics for evaluating low-level tasks, their practical meaning is more demonstrated in downstream high level tasks.

A1: We agree that high-level tasks are a vital part of AI for Life Science, but we do not agree this supports the conclusion that low-level tasks are subordinate to high-level ones. Purely low-level processing tasks command significant attention within the community. For eaxample, starting from early efforts like Content-aware image restoration (CARE) [1], which focused on fluorescence microscopy restoration and was published in Nature Methods in 2018, subsequent works such as BioSR [2] (Dataset & Benchmark), DeepLP [3] (Self-Supervision Learning), and UniFMIR [4] (Foundation Model) have repeatedly appeared in top-tier journals, each with a distinct focus.

From a presentation standpoint, a paper should have a clear focus. As we have emphasized since the title, our goal is to develop a versatile biomedical image processor, not an analyzer. Our work is benchmarked against low-level foundation models like UniFMIR in all aspects, including writing style, experimental setup, and validation methods. In these papers, using indirect downstream tasks to validate performance is not "usual" and can be less intuitive, as the choice of the downstream model can influence a fair comparison of the upstream processing effects. If you could specify which articles you would like us to compare against, we would be more than happy to make improvements.

Furthermore, while works focused on high-level tasks might emphasize downstream performance (e.g., the recent Cellpose 3 [5], which integrates a restoration module to aid segmentation), these are not the most suitable for a direct comparison with our model. Nevertheless, we performed an additional comparison using the same noisy dataset (68 images) from their work and fine-tuned our model on Cellpose3's training set. For further details on our model's performance on high-level tasks, we invite you to see our response to Reviewer S4FJ's Q7, which includes our fine-tuning results on segmentation and classification. We hope this statement clarifies the focus of our work. Thank you for your contribution!

Q2:

For each of the four tasks, evaluation is only performed on one or very small number of datasets. The evaluation on each task is far from being enough to show that the proposed method is generally applicable.

A2: We have included 13 datasets, covering over 30 baselines for comparison. At least 4 experiments are conducted for each task. For fairness, we directly compared with recently published works, adopting their experimental setups and data. Specifically:

• For restoration&super-resolution

Papers: UniFMIR (Nature Methods 2024), VCM (WACV 2025), and InverseSR (MICCAI 2023)

Datasets: CARE (Nature Methods 2018), BioSR-CCPs/ERs/Microtubules/F-actin (Nature Methods 2021), HBA (JNNP 2003).

• For registration&fusion

Papers: Transmorph (MIA 2022), ConvexAdam (TMI 2024), and BSAFusion (AAAI 2025)

Datasets: OASIS (MICCAI2021), IXI (MIA 2022), VIFB-CT&MRI/SPECT&MRI/PET&MRI (TMM 2023).

In total, the imaging modalities directly covered in our experiments include Light Sheet Microscopy (LSM), Multimodal Structured Illumination Microscopy (M-SIM), CT, SPECT, PET, and MRI, with an even wider variety of imaging content. Objectively, we have done our best to ensure our experimental volume is substantial within the limited space of a NeurIPS submission. Other reviewers have also acknowledged the breadth of our experiments (S4FJ: "Experimental efforts are comprehensive, with evaluation across many unique tasks, datasets, and modalities."; eeYn: "The effectiveness of the proposed method is validated through comprehensive performance comparisons with a variety of existing models."). If you could point out a specific data type or study you are particularly interested in, we would be glad to include it for validation in a future version.

Q3:

For the image fusion task, only three metrics are used to compare different methods, which is too small. Current image fusion studies typically utilize much more metrics.

A3: We would first like to clarify that the three metrics were calculated following the methodology of BSAFusion (AAAI 2025) and MSGFusion (TMM 2023), and we chose to display three key indicators for formatting and clarity. Since you did not specify which particular metrics or literature you would like us to include, we are providing additional metric results from our earlier experiments here for your reference.

Q4:

The training strategies of the proposed methods and the baselines are different in many experiments, which may make the comparison unfair.

A4: We respectfully disagree with this statement. First, we clearly list our source database, codebase, and baseline scores in Section A for full transparency. Second, for all experiments, our method uses a unified testing approach, not a changing one. This involves three fine-tuning strategies: fully fine-tuning, replacing the decoder, and replacing with a lightweight decoder. The detailed framework diagram and experimental setup are provided in Figure 7 and Section B. If possible, we would appreciate it if the reviewer could point out which sentence or section may have led to the misunderstanding of an unfair comparison, and we will be sure to avoid any ambiguity in future revisions.

Q5:

For the image registration task, there are a large set of different setting which bring different kinds of difficulty. Such as, registration between 2D images or 3D volumes, registration between different modality pairs or within the same modality, registration of different body parts with significant or not deformation, etc.

A5: Our model actually has both 2D and 3D versions, and we have already performed various registration tasks in our paper, including 2D/3D, atlas-to-patient, and patient-to-patient registration (Table 3 & 7, Figure 4 & 9). The data we use is inherently multimodal—in addition to the original MRI, it also includes its multi-labeled mask. Following the standard training procedure of works like Transmorph, we fine-tune on this task using two multi-labeled masks for supervised registration learning, while inference is performed directly on the two MRI images.

As requested, we are supplementing our results with an additional atlas-to-patient registration test on the multi-organ, four-dimensional extended cardiac-torso (XCAT) phantom data [6]. We have followed the same experimental setup as Transmorph, and the baseline results are taken from their paper. The final results are as follows:

XCAT-CT (16 organs)

Reference

[1] Weigert, Martin, et al. "Content-aware image restoration: pushing the limits of fluorescence microscopy." Nature methods 15.12 (2018): 1090-1097.

[2] Qiao, Chang, et al. "Evaluation and development of deep neural networks for image super-resolution in optical microscopy." Nature methods 18.2 (2021): 194-202.

[3] Fang, Linjing, et al. "Deep learning-based point-scanning super-resolution imaging." Nature methods 18.4 (2021): 406-416.

[4] Ma, Chenxi, et al. "Pretraining a foundation model for generalizable fluorescence microscopy-based image restoration." Nature Methods 21.8 (2024): 1558-1567.

[5] Stringer, Carsen, and Marius Pachitariu. "Cellpose3: one-click image restoration for improved cellular segmentation." Nature methods 22.3 (2025): 592-599.

[6] Segars, W. Paul, et al. "4D XCAT phantom for multimodality imaging research." Medical physics 37.9 (2010): 4902-4915.

We would like to extend our sincerest gratitude to the reviewer for the insightful feedback and high praise for our manuscript. We are truly honoured that our work is considered outstanding among many excellent submissions. A great deal of effort was dedicated to structuring the narrative, creating the figures, and designing the experiments, all with the goal of clearly and efficiently communicating the focus and contributions of our work within the limited space. It is incredibly encouraging to have our efforts recognized, regardless of the final outcome.

We have carefully considered all the comments and provide the following point-by-point responses:

Q1:

I originally misinterpreted the word “study”. In the context of medical imaging, this can often effectively mean “exam”; thus, “100 studies” seemed to be a very small amount. Context clues clarified this confusion quickly, but I would suggest potentially changing the wording here. There is probably a better way to clarify that this is an aggregate of many existing datasets.

A1: Thank you for this valuable suggestion. We used the term “study” because our primary data source, the Image Data Resource (IDR) [1], uses this term to describe the original user-uploaded source data. Currently, the resource contains 138 studies, which comprise 14,042,467 images and a total of 414TB of data. To prevent ambiguity, we will avoid using the term “study” when describing other data sources in our future versions of the manuscript.

Q2:

Related to the above point, I would try to highlight the “size” (e.g., # images) of the composite pretraining dataset as early as possible to convey the scale and dispel any confusion.

A2: This is an excellent point. We agree and will be sure to emphasize the scale of the dataset in future versions of the manuscript.

Q3:

Figure 1: While funny (to me, at least), I’m not sure “Crappifier” is appropriate.

A3: Thank you for your feedback. The term “Crappifier” was adopted from the Nature Methods publication, "Deep learning-based point-scanning super-resolution imaging" [2] which systematically investigates the suitability of various synthetic noise and artifacts for real-world microscopy image restoration and super-resolution. This paper was a significant reference for us when integrating self-supervised methods for these tasks. However, your point is well-taken; given that our work extends beyond image restoration and super-resolution, a more neutral term such as “Converter” would indeed be more appropriate. We will make this change in our revision.

Q4:

L172: Change “down-sample” -> “Down-sampling” L201 is not a complete sentence: “Resulting in more than 30…” Perhaps simply remove the period and write “resulting”

A4: Thank you for your meticulous reading. We will correct these in the next version.

Q5:

Table 5: The title is confusing, “Pre-train Strategies V.S Performance”. Should this read “Vs.” as in “versus”?

A5: Thank you for pointing this out. Your understanding is correct, and we will revise the title accordingly.

Q6:

Figure 5: Visually appealing, but difficult to read. Perhaps a different font/colour choice is needed, or the figure should be enlarged.

A6: We appreciate this feedback. The page limit constrained the current size of the figure for the ablation study. We will revise it for better readability in a future version of the manuscript where more space is available.

Q7:

I see no major weaknesses. However, I am curious to see how Orochi performs on downstream discriminative tasks like classification or even segmentation. (Weakness #1)

How does Orochi perform when fine-tuned on downstream (discriminative) tasks like classification? I understand that this does not fit under the umbrella of "low-level processing", but have the authors conducted such experiments? I would hypothesize it might perform well on dense tasks like segmentation, given its multi-scale pertaining. (Question #1)

A7: We conducted new tests on the model's performance on BTCV [3], REFUGE [4], and CadVidSet [5] segmentation datasets. We strictly adhered to the training hyperparameter settings of MedSAM2 [6] and carried out a complete fine-tuning following its GitHub repository. Baseline results are from the original paper.

Dice Score

Also, we conducted classification tasks on TissueMNIST (2D) and OrganMNIST (3D) datasets. All experimental setups are from the MedMNIST series papers [7] [8] [9].

We demonstrated the capability of Orochi to adapt to high-level tasks; however, we aim to avoid overstating its application. Our objective remains focused on the development of a unified low-level processor, as stated since the abstract of our paper.

Reference

[1] Williams, Eleanor, et al. "Image Data Resource: a bioimage data integration and publication platform." Nature methods 14.8 (2017): 775-781.

[2] Fang, Linjing, et al. "Deep learning-based point-scanning super-resolution imaging." Nature methods 18.4 (2021): 406-416.

[3] Landman BA, Xu Z, Igelsias JE, Styner M, Langerak TR, and Klein A, "MICCAI multi-atlas labeling beyond the cranial vault workshop and challenge," (2015)

[4] Orlando, José Ignacio, et al. "Refuge challenge: A unified framework for evaluating automated methods for glaucoma assessment from fundus photographs." Medical image analysis 59 (2020): 101570.

[5] Wang, Lu, et al. "Coronary artery segmentation in angiographic videos utilizing spatial-temporal information." BMC medical imaging 20.1 (2020): 110.

[6] Zhu, Jiayuan, et al. "Medical sam 2: Segment medical images as video via segment anything model 2." arXiv preprint arXiv:2408.00874 (2024).

[7] Yang, Jiancheng, et al. "Medmnist v2-a large-scale lightweight benchmark for 2d and 3d biomedical image classification." Scientific Data 10.1 (2023): 41.

[8] Yang, Jiancheng, Rui Shi, and Bingbing Ni. "Medmnist classification decathlon: A lightweight automl benchmark for medical image analysis." 2021 IEEE 18th International Symposium on Biomedical Imaging (ISBI). IEEE, 2021.

[9] Doerrich, Sebastian, et al. "Rethinking model prototyping through the MedMNIST+ dataset collection." Scientific reports 15.1 (2025): 7669.