One-to-Multiple: A Progressive Style Transfer Unsupervised Domain-Adaptive Framework for Kidney Tumor Segmentation

We would like to thank you for the positive consideration and the useful feedback on our work. We address all your concerns below.

W1. About Performance Enhancement.

The one-to-multiple UDA task presents many additional challenges compared to the one-to-one task. For example, the differences between various target domains are distinct, and a single generator must coordinate the complex mapping relationships among multiple target domains, thereby making the learning process more difficult. However, a single generator can translate one source domain into multiple target domains, significantly reducing the training time and resource costs compared to one-to-one methods. Overall, our findings have shown significant advantages and we believe our approach holds substantial benefits for multi-sequence UDA tasks.

W2. Scalability to more target domains.

As you suggested, we have acquired a new set of T1 MRI sequences from our partner hospital to validate the performance of our method in a one-to-five domain adaptive segmentation task. As shown in the table below, our method maintains high performance even with an increased number of target domains, which are derived from real-world clinical scenarios.

W3 and Q2. Abdominal multi-organ segmentation.

Due to the limited availability of medical data, we apologize that we were unable to collect a multi-target domain abdominal multi-organ dataset within a short timeframe. However, we collected a multi-organ dataset [1-2] consisting of CT and MR images from internet to validate the proposed method. The experimental results are presented in the table below.

We observe that although the results of MR to CT showed an improvement of approximately 42% compared to no domain adaptation, there is still significant potential for enhancement in comparison to fully supervised methods. Furthermore, the results from CT to MR were better. We attribute this difference to the higher image quality of CT in contrast to MR, which presents a greater challenge for domain adaptation from MR to CT.

Q1 Insights and future work.

As shown in Table 8 of Appendix E, the ablation study indicates that PIN brings significant performance improvements. However, the results for the multi-scale discriminator in Table 7 show that the improvement is not as pronounced. In our future work, we will conduct further investigation into different versions of discriminators. Furthermore, PSTUDA is essentially a multi-domain to multi-domain translation framework. We will also focus on developing a dedicated one-to-multiple UDA framework to achieve more substantial performance improvements.

Q3 Feasibility of deployment.

By utilizing the generator and multi-level style dictionary from PSTUDA in a clinical environment, we can achieve rapid one-to-multiple domain adaptation. We conducted simulations of PSTUDA’s performance using minimal computational resources with clinical data from our partner hospitals. Our PSTUDA requires approximately 13M of storage and 15G of computational resources. Thus, we believe that PSTUDA can be easily deployed even in clinical settings with limited computational infrastructure.

Q4 Code open source.

We will release the code, pre-trained models, and detailed implementation guidelines upon acceptance.

[1] Kavur, A. E., et al. CHAOS challenge-combined (CT-MR) healthy abdominal organ segmentation. MedIA 2021.

[2] Landman, B., et al. Multi-Atlas Labeling Beyond the Cranial Vault. 2017.

Thank you for your positive feedback on our manuscript. We're glad our efforts to address your questions were satisfactory. We will carefully consider your suggestion to move the ablation studies into the main paper.

As we mentioned in the main rebuttal, we include the answers to the additional questions of Reviewer tgyp. We hope that this helps to address all remaining concerns and we thank again for taking the time to review our work.

Q1. Overhead and warm-up for multi-level style dictionary.

In our experiments, the multi-level style dictionary essentially consists of a set of learnable tensors. Content is accessed through indexing, and as the style vectors are independent of each other, this does not result in any additional overhead.

Our method primarily focuses on adapting a source domain to multiple target domains simultaneously. All domains are fixed and visible during training, so there are no encounters with new unknown domains. Additionally, the multi-level style dictionary is randomly initialized at the beginning of training and does not require a warm-up phase.

Q2. Size of Multi-level Style Dictionary.

In our experiments, the multi-level style dictionary is directly created as a set of learnable tensors. Specifically, its dimensions are (4, 6, 4096), where 4 represents the number of domains, 6 represents the number of style fusion layers (four layers in the style fusion module and two layers in the decoder), and 4,096 represents the depth of the style dictionary. Therefore, this tensor contains a total of 4×6×4096=98,304 parameters, which converts to approximately 0.10M.

Q3. Generator and Discriminator Architecture.

• Generator: PSTUDA uses the CycleGAN generator. The difference lies in the replacement of normalization operations in the style fusion module and decoder from AdaIN to our proposed PIN. Additionally, we employ a progressive style injection method layer by layer to achieve image translation.

• Discriminator: PSTUDA’s discriminator architecture is based on the multi-scale discriminator from MUNIT. The difference is that we replace the Conv2dBlock in MUNIT with ResBlk, which has a residual structure. Additionally, the multi-scale output sizes are set to 1/16, 1/32, 1/64, and 1/128 of the original image size. Our motivation for using a multi-scale discriminator is that the generator, integrated with PIN, will have enhanced generative performance. Therefore, the discriminator also needs to be more powerful to match the generator, enabling better adversarial training and effectively leveraging the generator’s capabilities.

[1] Kavur, A. E., et al. CHAOS challenge-combined (CT-MR) healthy abdominal organ segmentation. MedIA 2021.

[2] Landman, B., et al. Multi-Atlas Labeling Beyond the Cranial Vault. 2017.

[3] Huang, X., et al. Arbitrary Style Transfer in Real-time with Adaptive Instance Normalization. ICCV 2017.

[4] Nam, H., et al. Batch-Instance Normalization for Adaptively Style-Invariant Neural Networks. NeurIPS 2018

We greatly appreciate your constructive and insightful comments. We address all weaknesses and questions below.

W1. Open-source code is not explicitly provided.

We will release the code, pre-trained models, and detailed implementation guidelines upon acceptance. To ensure reproducibility, we build on the open-source implementations of StarGAN v2 and CycleGAN, and provide all relevant hyperparameters in Appendix B.

W2. One-to-Multiple framework needs more details.

We apologize for this and will provide a more detailed description of our One-to-Multiple PSTUDA framework in subsequent versions.

W3. Bidirectional validation on cross-modal MRI and CT data.

According to your suggestion, we conducted bidirectional cross-modal validation on the abdominal multi-organ dataset [1-2] and performed reverse validation experiments from MR to CT on the MSKT and KiT19 datasets. The results are shown in the following two tables. Both sets of experiments demonstrate that our method significantly outperforms StarGAN v2 (baseline).

Q4. Theory of PIN.

We believe that PIN has significant advantages in certain complex scenarios. For example, in kidney tumor medical imaging data, the contrast, texture, and morphology of the lesion areas exhibit significant differences from normal tissues. These differences ultimately stem from variations in data distribution. In such cases, PIN customizes a set of scaling parameters for each local spatial point in the content feature map for each channel, allowing for more precise alignment of local differences, thereby making the translated images closer to the real target domain’s data distribution. The calculation formula for PIN is:

where $\gamma$ and $\beta$ are obtained through convolutional transformations via $v_{ss}$, and their dimensions are consistent with those of $\hat x_{nchw}$.

Q4. Effectiveness and Interpretation of PIN.

We conducted a comparison of PIN with the widely used AdaIN [3] and BIN [4], as shown in the table below. The results show that our PIN achieves the best performance. Theoretically, fully supervised segmentation results should be the upper limit for UDA, as it is conducted within the same domain, with both the training and test sets belonging to the same distribution. The key challenge in UDA tasks lies in addressing the differences in data distribution between different domains. Hence, the closer the distribution of the translated images is to the real target domain’s data distribution, the better the segmentation performance. Thus, we believe that the precise style adjustments in PIN will further reduce data distribution differences, leading to more accurate target segmentation.

Additional comment: We would have liked to include the rest of answers to the questions mentioned by Reviewer tgyp (marked as Q1,Q2, and Q3). Unfortunately, we did not have enough space in this rebuttal box. As soon as the discussion phase will begin, we will include the mentioned answers in an additional comment for the reviewer.

As we mentioned in the main rebuttal, we include the answers to the additional questions of Reviewer yJwR. We hope that this helps to address all remaining concerns and we thank again for taking the time to review our work.

W3. Ablation Study on key hyperparameters affecting model performance.

Our main contributions lie in the multi-level style dictionary and the style fusion module. These two modules have the following hyperparameters: the depth of the style dictionary, the number of levels in the style dictionary, and the number of style fusion modules. Note that the number of levels in the style dictionary is equal to the number of style fusion modules.

To investigate the sensitivity of model performance to various hyperparameters, according to your suggestions, we conducted extensive ablation studies on the depth of the multi-level style dictionary (MSD) and the number of style fusion modules (SFM). The results, presented in the table below, indicate that under the original settings (dictionary depth of 4,096, module number of 4), PSTUDA performs optimally in most experiments. Notably, when we extended our dictionary depth to 16,384, there was an improvement in segmentation performance on T2W sequence.

[1] Huang, X., et al. Arbitrary Style Transfer in Real-time with Adaptive Instance Normalization. ICCV 2017.

[2] Nam, H., et al. Batch-Instance Normalization for Adaptively Style-Invariant Neural Networks. NeurIPS 2018.

We greatly appreciate your constructive and insightful comments. We address all weaknesses below.

W1. Practical and clinical relevance of PSTUDA.

PSTUDA is designed as a one-to-multiple UDA framework based on multi-sequence MRI segmentation tasks, thus having strong clinical relevance. Compared to methods that train target domains sequentially (including continued UDA and UDA on evolving domains), PSTUDA offers the following advantages:

• Flexibility: Sequential methods treat all target domains as a single domain during training, which prevents them from generating images of the specified target domain during inference, potentially leading to inaccurate results. In contrast, PSTUDA can generate images of the specified target domain as needed during inference. This flexibility meets the diverse requirements of clinical applications.
• Training efficiency: Sequential methods train in a serial manner, whereas PSTUDA trains all source and target domains in parallel, significantly reducing training time. Additionally, PSTUDA’s generator and multi-level style dictionary enable quick deployment and fast inference, meeting clinical demands.

W1. Novelty of PSTUDA.

PSTUDA is inspired by OMUDA and StarGAN v2. However, we believe there are some fundamental differences between our method and these models:

• Style feature acquisition: OMUDA extracts style features from a style encoder or mapping network, which lacks representativeness and stability. In contrast, PSTUDA addresses this issue by using a multi-level style dictionary to directly learn the global style features of each domain.
• Generator architecture: Both OMUDA and PSTUDA use the CycleGAN generator. However, OMUDA uses the same style features in each layer of AdaIN, while PSTUDA replaces AdaIN with PIN and progressively injects different style features layer by layer to achieve image translation.
• Discriminator architecture: OMUDA uses the StarGAN v2 discriminator, whereas PSTUDA employs a multi-scale discriminator to better leverage the capabilities of its powerful generator integrated with PIN.

The ablation experiments in Appendix E demonstrate the effectiveness of our method.

W2. Evaluation of PIN effectiveness.

AdaIN [1] and BIN [2] provide global scaling parameters for each channel of the content feature map. In contrast, our PIN offers unique scaling parameters (mean and standard deviation) for each local spatial point in each channel.

We believe that PIN is particularly valuable for fine-grained segmentation tasks due to its ability to consider local style differences, enabling PSTUDA to generate synthetic images that better match the target domain data distribution.

To demonstrate our perspective, we conducted ablation experiments on the three methods, as presented in the table below. The results indicate that PIN outperforms AdaIN and BIN in terms of performance, especially in kidney tumor segmentation. This superiority can be attributed to the fact that kidney tumors, being abnormal pathological tissues, exhibit significant style differences which can be finely processed by PIN.

Additional comment: We would have liked to include the rest of answers to the questions mentioned by Reviewer yJwR (marked as W3). Unfortunately, we did not have enough space in this rebuttal box. As soon as the discussion phase will begin, we will include the mentioned answers in an additional comment for the reviewer.

We are pleased to have addressed most of your concerns and we sincerely appreciate you raising our paper's score to 5: Borderline Accept. Your feedback has been invaluable to our work, and based on your suggestions, we will include the new supplementary experimental results in the revised version. Thank you for your support！