Synthetic Data as Validation

• Appendix D: can you put a rectangle for the generated tumors? This will help reader to spot it easier.
• Appendix D: Can you show real examples of tumors, so that we can compare the synthetic ones and real cases?
• Figure 2: what are the different grey lines? I see that you mention: The curves generated by the real-tumor validation are plotted in gray. But what does each line correspond to, or why do you have multiple grey lines?
• Figure 2: how does the dice score look like for training data?
• Do you have ablation study on number of synthetic validation data? Does it play a role?
• How the tumors are generated seem not to be well studied. Do you have ablation study on this? E.g. is there other ways of texture generation, does the boundary of the tumor play a role, is it always a sharp contrast etc.?
• Section 5.4: how does the tumor count calculated? More information for sensitivity analysis is needed.

Minor:

• Typo on p3: “…We continuously create a stream of synthetic data…” instead of creates.

• The only comparison to validation with real data is a small dataset with 5 volumes, which is difficult to use as a proper baseline. To have actual value of a study on synthetic data for validation, it would be important to understand the interplay between validation set size and test performance. How large val size is needed for robust model selection? How does validation on real vs. synthetic compare at equal validation set size? Would augmentations on real data in validation set be equally valuable compared to adding synthetic data? How about combining real and synthetic data?

• Experiments only test model selection over training length, which requires there to be sufficient variations (e.g. over-fitting) during training. For thorough testing of validation set for model selection, there should also be experiments on a selection of other hyper parameters (e.g. learning rate, architecture selection, augmentation on training data, etc.).

• Figure 2 is the only result supposed to demonstrate the weakness of selection based on the real validation data, but this is rather weak. What are the different curves of real validation? Is this for different models or different volumes ("variations observed when different validation samples are chosen")? Is model selection based on results of single volumes? This does not make sense, and is expected to be very sensitive to the characteristics of individual volumes. Averaging of all validation data should be used for model selection. Also, are these curves averaged over multiple trainings?

• The continual learning setting seems unclear and artificial. For this purpose, a static dataset (cohort 4) is used. In what way does this represent continual learning?

• The results in Fig. 5 mostly demonstrate benefits for training on synthetic data, not validation. As such, these results doesn't contribute to the main aim of the paper of introducing synthetic data for validation.

• Ultimately, the use of synthetic validation data is only tested for one type of data and one synthetic data generation strategy. It is not clear how this would generalize to other tasks, data, and synthetic data generation. As such, the impact of the results is rather weak.

• It would be easier to interpret the generated data in Fig. 11 if segmentation maps were also provided.

• The synthetic tumor generation seems to require quite an amount of manual labor. Would it be possible to use generative modeling for generating tumors, conditioned on, e.g., tumor size and location?

• The term "out-domain" is used throughout the paper, whereas "out-of-domain" or out-of-distribution (OOD) would be more appropriate in relation to the literature.

• It is somewhat confusing reading the introduction, motivating the use of synthetic data for validation, when it suddenly also discusses using synthetic data for training, for example: "We find that synthetic data can facilitate a more reliable performance estimate on unseen data and effectively address the constraints commonly associated with small, biased validation sets. Specifically, we synthesize tumors in the healthy liver, which gives us orders of magnitude larger datasets for training". This could be clarified.

Please clarify the creation of the tumor model, include guides of where synthetic tumors are present in CT, and provide a comparison of the proposed tumor addition strategy with standard data augmentation methods.

Questions

1. Could the authors please explain why the DSC is so low (around 30%) and why it is much lower than that of most methods tackling the same task (see Weakness #5 for further explanations)
2. Could you please explain the rationale for choosing such a very small (5 samples) real validation dataset which is not a realistic setting for many applications and the rationale for not experimenting with varying validation set sizes?
3. Please clarify if the metric used in Section 5.4 is the “voxel-wise” sensitivity. If that’s the case, it is not adapted to assess detection and should be replaced with “lesion-wise” detection metrics. Also, analyzing sensitivity alone is not sufficient.
4. Please clarify the continual learning procedure, its dynamic aspect, how synthetic tumors are dynamically tailored to align with emerging distributions and if/how forgetting is assessed/dealt with.
5. The paper mentions 100 CT volumes for the LiTS dataset while the LiTS challenge website mentions that 130 CT scans are made available to the participants for training. Could you please explain the difference?

Comments (I added the following comments because I think they may be useful to the authors for improving the paper but they don’t call for a rebuttal).

1. Even if the paper were to make a convincing case that using synthetic data as validation leads to improvements (which is not the case currently), the title is too general as the paper only studies this for a particular task and a particular type of data. It is perfectly understandable that it is not possible to study many tasks and different data types in a conference paper but then the paper title should reflect this, e.g. “Using synthetic data as validation set for medical image segmentation”
2. There are dozens of variations of the U-Net. Please specify which U-Net architecture is used and provide the exact reference (the reference currently given is the original 2D U-net and I doubt this is the architecture that was used in the present paper). I suggest to provide the exact reference and to describe the architecture in details in the appendix.
3. Please provide confidence intervals for all reported metrics, not only DSC.
4. I don’t agree that one of the factors, mentioned in footnote 1 page 2, explaining greater utility of data synthesis in medical imaging compared to CV is that “Firstly, the focus is primarily on synthesizing tumors rather than other components of the human anatomy.” On the contrary, I believe that papers such as SynthSeg (Billot et al, 2023) (B Billot, DN Greve, O Puonti, A Thielscher, K Van Leemput, B Fischl, ..., SynthSeg: Segmentation of brain MRI scans of any contrast and resolution without retraining, Medical Image Analysis, 102789) have shown the opposite. On the other hand, the second mentioned factor, although speculative (tumor variability is debatable) seems ok to mention.
5. I think the work by Billot et al (2023) should be mentioned in the related work as this is a good example of successful training using synthetically generated medical images that produces variations of the anatomy and the image contrast (rather than variation of a lesion such as a tumor).
6. It seems that the tumor generator described in Section 3.2 is exactly the same as that presented in (Hu et al, 2023) however this is not explicitly said in Section 3.2. The introduction suggests that it is (“we employ a modeling-based strategy (Hu et al., 2023)”) and by comparing this reference to Section 3.2, they seem similar. If the authors are directly using the generator proposed by Hu et al (2023), please state clearly in Section 3.2 that the procedure is identical and that this section is simply a summary of this procedure. On the other hand, if the two are not identical, please explain the differences.
7. Please clarify that tumor characteristics that serve for the generation come only from the LiTS dataset (Appendix A suggest so but it would be better to be more explicit).
8. Statistically speaking, I don’t think one can say the real-tumor validation dataset is “biased” (Section 5.1). The problem here is the variability of the sampling process, but the sampling process itself is not biased (it is just that out of the validation samples that may be selected, some may be very far off the testing distribution).
9. Do not say “significantly” unless it is statistically significant. If you want to say that there is a strong difference, you can say “substantial”.
10. I suggest to clarify if some samples of LiFT have been left out and why. Also, for reproducibility of the paper, it would be good to include the exact list of samples from each dataset that were included. For FLARE’23, this would have the added benefit to ensure that there was no cherry-picking during the selection.

Minor suggestions

• Please always say “validation set” and not “validation” since validation can refer to the whole validation procedure (data splits, metrics…)
• p1, abstract: I don’t think “rigorous” validation is the right wording (since here validation refers to validation set, not to the overall validation procedure). I suggest to replace with “robust” or something like that.
• “out-of-domain” seems grammatically more correct than “out-domain”
• I think one should not say “the model… may not be generalized” but rather “the model… may not generalize”
• Page 4, last paragraph: when referencing Ronneberger et al, 2015, the authors are specifically referring to the elastic deformations used for data augmentation in this paper but this could be made clearer.
• Figure 7: please define HU: Hounsfield unit
• Figure 11: please also display the labelled tumor (superimposed on the CT scan) as an additional panel below the image
• Section 4.2: please mention which loss was used

• What do texture and intensity imply in the context of tumors?
• Considering that only 120 CT volumes remained, how diverse is cohort 7?
• Apart from tumor size, were there any other differing factors in the healthy CT assembly? Is this cohort representative and diverse?
• In Figure 2, why are there multiple lines for validation? Were different validation samples selected, and if so, how?
• In Figure 3, it is challenging to make a direct comparison visually with Figure 2. Unsure which experiment corresponds to which number in Table 2. What data were used for training here?
• In Figure 5, sensitivity was shown but did the authors look at false positives?
• In Figure 6, it appears that the out-domain had a higher DSC than the in-domain. Were the test results were based on in-domain or out-domain synthetic-tumor validation?
• How high is the computational cost?