PIE: Simulating Disease Progression via Progressive Image Editing

Weaknesses 1's reply: One insight of PIE is even if it is difficult to obtain real disease progression from the same patient, it is still possible to use all data across different patients to build an unified model to predict disease. Simulating disease progression in images without direct temporal data relies heavily on the learned distribution of generative models, the quality and diversity of the training data, domain knowledge, and iterative refinement processes. Generative models like diffusion models are often used to simulate realistic images. These models can learn complex distributions from a dataset and generate new instances that are statistically similar to the observed data. In PIE, text acts as a locator of which part of implicit distribution the inference step would use. Even without temporal data, diffusion models can still learn correlations between disease stages and specific visual features.

Weaknesses 2's reply: We propose a framework to solve the medical imaging problem with plugged in generative models. DDIM happens to have certain nice properties of convergence as well as state of the art generation quality on images.In addition, it’s also easier to train and adapt given our limited resources. We could technically swap in other text-image generative models, as long as they share the same convergence properties as DDIM.

Weaknesses 3's reply: Actually, data visualization is better than CLIP-I for fidelity evaluation. This is because medical images have a domain gap to the general domain. As shown qualitatively in figure 4 and figures in appendix, images produced by earlier steps tend to be closer to real image distribution, which happens across all three methods. This tends to average out our advantage in CLIP-I scores. However, the quality of the generated images would evidently show that the improvement in CLIP-I scores are definitely NOT marginal.

Question 1's reply: Diabetic retinopathy dataset is the easier one in the selected datasets. CheXpert contains over 100K imaging data for 6 classification and containing co-occurrence diseases. Kaggle Diabetic Retinopathy Detection Challenge contains over 40K imaging data for 2 classification and ISIC 2018 contains about 10K data for 4 classification. When datasets contain more co-occurrence disease cases and less data, it will be difficult to pretrain generative models.

Question 2's reply: Yes, as with all generative models, it’s pruned to hallucinate, but we believe with larger and better curated datasets, this problem can be overcome. We hope this paper can serve as the starting point and provide the community the motivation and incentive to get the ball rolling on disease progression simulations.

Question 3's reply: Thanks for pointing out. We will move the ROI mask ablation study to the main body.

Weaknesses 1's reply: we conduct experiments to predict the potential disease future of individual patients. In the ablation study, we compare PIE generated results with real longitude medical imaging sequences. In order to validate the disease sequence modeling that PIE can match real disease trajectories, we conduct experiments on generating edema disease progression from 10 patients in BrixIA COVID-19 Dataset (Signoroni et al., 2021). The input image is the day 1 image, and we use PIE to generate future disease progression based on real day 1 clinical reports for edema. The 10 generated and the real data pairs from Figure.7 have been checked by clinical doctors with 80% agreement.

Weaknesses 2's reply: Thanks for pointing out this misunderstanding sentence. For CheXpert, EHR (including future treatment) are shown in part of the report. For ISIC 2018, Kaggle Diabetic Retinopathy Detection Challenge, the report doesn't contain therapy information. We will rewrite this sentence to show the author only part of datasets can predict future treatment responses in image space.

Weaknesses 3's reply: It can be personalized by using the patients’ medical records as text instruction and an initial medical image as input for PIE.

Weaknesses 4's reply: We would argue otherwise, our experiment results show that in the end it can generate realistic images with verifiable disease features without diverging to human unreadable noise like other methods do. We believe that itself is the best evidence. Our ablation study in the appendix also shows that our method converges in around 10 steps, which is fast and practical number of steps to solve the problem.

Weaknesses 5's reply: A well-experienced radiologist can predict disease progression based on their understanding of CT, X-ray, MRI imaging. However, they will not make deterministic statements.

Weaknesses 6's reply: The reason is: the input patient's X-ray image and clinical report is a state that has already been diagnosed with that disease, but not achieved a serious condition. Thus, we used a proposed model to simulate the potential changing sequence. We also conducted comparative experiments using real longitudinal data, and the results showed that the confidence score of the PIE-generated image and the judgments of the three clinicians were close to the real progress. Previous work must use large amounts of longitudinal data, which are, however, almost impossible to obtain.

Weaknesses 7's reply: Thanks for your suggestion. We will adjust the wording accordingly. “personalized healthcare in the imaging space” means the current datasets don't contain each patient’s long-term history data (MRI, X-ray, CT,...).

Weaknesses 8's reply: Denoising Diffusion Implicit Models.

Weaknesses 9's reply: We discussed the details of proposition 1 in the appendix section B1, including the definition of “learning rate”. We show that $\epsilon^{(i)}\_{\theta}(x_1^{{(i)}}, y) := \nabla_{x} \log p(x | y)$. This can be seen as gradient descent with a geometrically decaying learning rate with a factor of $\sqrt{\alpha_0}$, with "base learning rate" $\frac{\sqrt{\alpha_1 - \alpha_0\alpha_1} - \sqrt{\alpha_0 - \alpha_0\alpha_1}}{\sqrt{\alpha_{1}}}$.

Weaknesses 10's reply: The purpose of proposition 2 and 3 is to show PIE will lead to a stable convergence, as we claimed in the paper: “the changes between steps would grow smaller”. We apologize if variables are not self-explanatory enough in the propositions, we will add more text for clarifications in the later versions.

Weaknesses 11's reply: As stated in the assumption of proposition, we since we assume the denoising step generated is bounded by a constant. Hence the only thing determining the scale of changes is the step size, which is very similar to “learning rate” in SGD.

Weaknesses 12's reply: They are the confidence scores and the CLIP-I scores which measure how good the generated images are based on the ground truth.

Weaknesses 13's reply: Half of the medical doctors and radiologists only finished the survey and did not provide the reason. Here are some key questions from the other doctors who feel strange about the progression: (1) no co-occurrence disease in the image; (2) pacemaker disappear. We have discussed these issues in the appendix.

Weaknesses 14's reply: The real world standards mean the real disease progression verified by multiple experienced medical doctors. This might not be the golden standard but that is the best we can get. In the future, it can be improved by more clinical professionals.

For Question: We would like to express our sincere gratitude to Reviewer KqFr for pointing out these weaknesses and giving us potential improvement directions. Yes, this method is used for the disease prediction for future time points and we have proven it can be useful to predict Edema in existing COVID dataset.

Weaknesses 1's reply: We thank you for your suggestion. However, text condition is not the major part of the proposal as it is the basic structure of any stable diffusion-based generative models such as DreamBooth and ControlNet for image generation. These articles generally do not conduct ablation experiments on text conditions as these text inputs are encoded with frozen CLIP text encoder. As you suggested, we conducted an extension experiment based on testing different text conditions in the MIMIC-CXR dataset for PIE's simulation results. For experiment, we only use the test set and the simulation steps are 10 for each disease sequence.

• MIMIC-CXR-PRO addresses the issue of hallucinated references to priors produced by radiology report generation models.

Weaknesses 2's reply: Thanks for pointing this out! Since the longitudinal medical data is scarce, we only chose a group of data with longitudinal information to do the validation of our method. It is impossible to generate similar medical images due to the environmental and sensor noise (our purpose is to provide similar clinical information to completing patient clinical longitudinal data and predict the future disease progression direction). The 10 generated and the real data pairs from Figure.7 have been checked by clinical doctors with 80% agreement.

Weaknesses 3's reply: You raise an important point regarding the potential applications of our method, and I appreciate the opportunity to clarify further. Indeed, the current limitations due to the scarcity of longitudinal data cannot be overlooked. However, the promising results from generating future disease images support the method's potential beyond medical education. Combining with techniques like human-in-the-loop active learning, we anticipate that our method will progressively refine its capability to model disease trajectories, enhance prediction of treatment responses, and competently fill gaps in clinical records. The journey to these goals is indeed a long one, but it is feasible with the incremental integration of data over time, which needs the effort of the whole community.

Question 1's reply: In clinical medicine, we seldom say deterministic disease progression but alway say potential disease progression. In non-image state-based generated disease progression research, the development direction of the patient's disease is generally predicted by a probabilistic model, and deterministic results are also not generated. On the contrary, the randomness can support clinical researchers understanding potential change for each disease and enhance the model's capability to simulate co-occurrence diseases based on the context and image space.

Question 2's reply: No. The input image is healthy while the health report is from the patient who is in the middle of certain diseases. Our method demonstrated that the generated trajectories can follow the progressive trend of the patients’ medical histories and provide clinical values.

We would like to express our sincere gratitude to Reviewer SCY9 for the insightful remarks and for dedicating their time to reviewing our paper. We will follow your general remarks to improve both the figures and contents.

Question 1's reply: 1-a. In general, the available imagery per patient is 1. only part of patients in the dataset contain longitude data sequence (>=2 imagery from different times). 1-b. They are not in the same time frame. 1-c. The disease annotation for each patient is uniform. We can't say disease stage is uniform because the text input is a clinical report. It contains disease stage description.

Question 2's reply: We randomly placed (shuffle) the generated progress and the real progress in the questionnaire, which was analyzed by the participating doctors. Thus, they don't know which one is the generated evolution image.

Question 3's reply: In our follow up survey, the generated imaging sequences are reviewed by 3 experienced radiologists. They carefully analyzed each image and, based on clinical experience, believed that the generated disease progression images have good clinical consistency.

Question 4's reply: 4-a, for ISIC 2018, Kaggle Diabetic Retinopathy Detection Challenge, we use only one (Image, text) pair per patient. For CheXpert, each patient may contain more than one (Image, text) data pair, but we shuffle the training set so they will not leak longitude information. 4-b This is because most of the current datasets don't contain real longitude imaging data sequences for each patient. One of our goals is, can we simulate imaging disease progression close to clinical reality without using longitude imaging data sequence in the training data.

Question 5's reply: Before disease change estimation, we use \alpha as a group of control params to add noise to the input X-ray image $x_{0}^{(n-1)}$. The denoising inference using DDIM is a multi-step process. In each step, the stable diffusion unet $\epsilon_{\theta}$ is used to denoise the input using a conditional clinical report and then use a non-linear differential equation defined by \alpha. More detail is shown in Appendix A BACKGROUND and B.1 PROOF OF PROPOSITION 1. DDIM is the single step disease progression estimation of PIE.

For multi-step disease trajectory simulation, we run the DDIM multiple times with the control params $\beta_1$ and $\beta_2$. These two params are used to balance the editing and unchange interpolation. We introduce them in the 5.1 EXPERIMENTAL SETUPS.

Question 6's reply: It is a global performance for the proposed method. Actually, even real disease progression can only match physicians’ expectations with 80-90%. Clinical medicine is a discipline that values experience. Even so, the 76.2% match rate still reflects the effectiveness of the proposed method.

Physicians’ Expectations = $\frac{1}{N} \sum_{i=1}^{N} ( \frac{Conf_i}{5}) * Agreement_i$

where $Conf_i$ is an integer between 0 to 5. $Agreement_i \in \{0, 1\}$ is a binary boolean value.

Question 7's reply: Only one report was used for a patient during training or inference.

Question 8's reply: 8-a. Yes, the machines generated for synthetic reports are also reviewed by 3 experienced medical doctors. 8-b. During training, we all used real reports. During inference, we test both real and synthetic reports for each disease, the real : synthetic ratio in CheXpert is 1:5, while in ISIC 2018, Kaggle Diabetic Retinopathy Detection Challenge, the ratio is 1:1. The reason is in CheXpert, there are 6 diseases for generation, but each patient in the test set only contains the real report for one disease. 8-c. For all datasets, the real reports are in the same style.

Question 9's reply: The refinement of the image is to train a model that can understand both the semantic of the image and clinical context. Yes, the proposed model can generate the predicted image-based disease progression sequences.

Question 10's reply: The key semantic features are disease related areas in the input image. For example, for skin imaging, it is the regions that contain skin lesions.

Question 11's reply: For CheXpert, the medical conditions are the therapy and complications shown in the report. For ISIC 2018, Kaggle Diabetic Retinopathy Detection Challenge, the report doesn't contain therapy and complication information, so medical conditions are not included. Thanks for pointing it out, we will rewrite this sentence to show the author which dataset uses additional medical conditions.

Question 12's reply: Thanks for pointing out. What we want to express is that PIE can predict the disease progression only considering the information provided by the input clinical context, and will not produce false positive disease features.

Question 13's reply: Yes. CheXpert is the largest which contains over 100K imaging data. Kaggle Diabetic Retinopathy Detection Challenge contains over 40K imaging data and ISIC 2018 only contains about 10K data.

Question 14's reply: Yes.