Thanks for your insightful comments and suggestions. Below we will address all the questions and concerns.

Re adding reference image in Figure 4:

Thanks for the suggestion. We have the reference image in Figure 5, second from left on top row. We will add it to Figure 4 in the final version.

Re GPT-4 evaluation:

Although GPT-4 has no specialized training in biomedicine, many prior studies have demonstrated its impressive capabilities in understanding and processing biomedical text. E.g., book by Peter Lee et al. on: “The AI Revolution in Medicine: GPT-4 and Beyond.” offers insights into this aspect.

Additionally, one of the coauthors, a board-certified practicing radiologist at a top US medical school, has played a crucial role in assessing the utility of GPT-4 in our context. The progression description generated by GPT-4 appears reasonable upon preliminary examination. We will add more systematic assessments in the final version.

To illustrate, here are some examples reviewed by the radiologist, comparing GPT-4 vs orginal text:

These examples demonstrate GPT-4's ability to succinctly summarize key radiological findings, which aligns well with clinical needs and workflow efficiency. This aspect will be further elaborated in the final version of our manuscript.

Likewise, in addition to the quantitative evaluation, we have conducted several review sessions with the aforementioned radiologist co-author, who expressed satisfaction with the image generation quality. We will mention this in the final version. Thanks much for your suggestion.

Re counterfactual vs image editing:

We agree that the underlying technique might share a lot of similarity, but the real-world semantics are quite different. In biomedicine, a clinician won’t be thinking about this generation problem as editing an image. Instead, their interest lies in counterfactual reasoning, i.e., projecting what if the patient’s condition has progressed in certain ways. Many prior works have studied the “Counterfactual” factor for medical images, e.g. Cohen et, al. “Gifsplanation via Latent Shift: A Simple Autoencoder Approach to Counterfactual Generation for Chest X-rays”; Sumedha Singla, et, al. “Explaining the Black-box Smoothly- A Counterfactual Approach”. In this context, counterfactual just means answering what if questions, which could be with or without the prior image. However, in the case of progression modeling, the patient already has some prior image taken. So it is natural to condition on the prior image, rather than generating a newmage from scratch without any constraints. This distinction is pivotal in our research, as it aligns with the clinical workflow and decision-making processes, emphasizing the practical applicability in real-world medical settings. We will mention this in the final version.

Re Table 4:

Thanks for the insightful comment. Here, we are comparing the similarity of a generated image with the prior image. The reference image may have lower DICE score due to variations in the creation process for the consecutive images, which are taken at different times, potentially by different clinicians with different machines. This variation is a typical characteristic in clinical imaging and does not imply inferiority of the reference image.

By contrast, the extremely low scores for baseline systems indicate that they are completely off base with respect to the prior image.

We will clarify this in the final version.

Thanks for your insightful comments and suggestions. Below we will address all the questions and concerns.

Re comparison to Imagic:

Thanks for this suggestion. Imagic requires per-example inversion and fine-tuning. Namely, for each input image and target description, it requires a fine-tuning process to obtain an optimal text embedding close to the target description, fine-tune the diffusion model conditioned on the optimal text embedding (rather than noise in the original diffusion model), and then apply the newly fine-tuned diffusion model to an interpolation of original & target text embeddings to generate the new image. This is very cumbersome and not very generalizable.

By contrast, our method builds on the more recent InstructPix2Pix method, which learns a single model that can generate new image in a single forward pass, without requiring elaborate text embedding and diffusion learning for each test example as in Imagic.

We will add discussion to explain why our method is superior to the Imagic approach.

Re reader study by radiologist:

One of the coauthors is a board-certified practicing radiology at a top US medical school. In addition to the quantitative evaluation, we have conducted several review sessions with this radiologist co-author, who expressed satisfaction with the generation quality. We will mention this in the final version. Thanks much for your suggestion.

Re ethical considerations and limitations:

Thanks much for this insightful suggestion. We have discussed limitations in the Appendix. But as you suggested, this is such an important aspect that we will move our existing discussion on limitations to the main body and add additional considerations for ethical implications.

Re generalization to other medical image datasets:

Thanks much for this suggestion. While we tried to break new ground in counterfactual medical image generation, this is just the first baby step towards a universally applicable tool. We plan to explore exactly along the directions you have outlined in future work.

Re citation:

Thanks for the good catch! We will fix them.

Re demographic diversity:

Thanks much for this insightful suggestions. We will move our discussion about demographic diversity to the main body. Further, we have conducted additional evaluations focusing specifically on racial diversity. Our analysis compared image generations for the dominant demographic group (white) and an underrepresented group (black). Despite the training data itself being skewed in terms of racial representation (17% black vs 78% white), our results are promising. MedJourney successfully retained the racial features of the prior image with high fidelity, scoring 0.92 for the underrepresented group and 0.98 for the dominant group (we used the SOTA image race classifier, as mentioned in sec 4.3, as evaluator and AUC as metric). This indicates our MedJourney has a decent property of preserving demographic diversity, and further suggests that it may contribute to mitigating racial representation disparities through data augmentation for underrepresented groups. While our results are promising, they also emphasize the crucial need for balanced datasets in medical research. We will include this insight in the final version.

Re adding FID metric:

Thanks for the suggestion. We have incorporated this metric into our analysis and will present it in the final version. The FID results are as follows:

We believe these results further demonstrate the effectiveness of MedJouney.

Thanks for your insight comments and suggestions. Below we will address all the questions and concerns.

Re concerns with text processing:

Validity of disease progression: Thanks for the suggestion. One of the coauthors is a board-certified practicing radiology at a top US medical school. The progression description generated by GPT-4 appears reasonable upon preliminary examination. We will add more systematic assessments in the final version. In general, although GPT-4 has no specialized training in biomedicine, many prior studies have demonstrated its amazing capabilities in understanding and processing biomedical text. E.g., see the book by Peter Lee et al. on: “The AI Revolution in Medicine: GPT-4 and Beyond.”

Lack of sanity check: In addition to the quantitative evaluation, we have conducted several review sessions with the aforementioned radiologist co-author, who expressed satisfaction with the generation quality. We will mention this in the final version. Thanks much for your suggestion.

Comparison between Impression and GPT-4: The impression section is taken from the gold report describing the target image. The small difference between using impression vs GPT-4 generated progression may stem from the characteristics of this dataset, where in many cases the first image is normal, and thus the pathological description of the follow-up image essentially describes progression. Additionally, some impression would include comparison against the prior image, also similar to what GPT-4 would generate as progression. As in 5.3 “impression v.s. GPT-4 generation”, GPT-4 generated is usually more succinct (averaging 146 characters v.s. 179) and more naturally phrased. For example,

Re image registration contribution:

Note that image registration is only a preprocessing step that helps align the two consecutive images for a given patient in the training data. This is a standard technique to reduce immaterial variations of images taken at different times by different technicians, e.g., translational shift or rotation. Image registration alone can’t perform counterfactual image generation and is thus irrelevant with respect to the novelty of MedJourney. We will clarify this in the final version.

A significant challenge in our research was the limited dataset size (approximately 10,000 triplets), posing the risk of learning spurious correlations due to minor, non-essential variations in images. By implementing image registration, we effectively minimized these immaterial variations, enabling the model to focus on material changes indicative of disease progression.

This image registration procedure is used merely for preprocessing training data. This procedure is illustrated in Appendix Figure 8, and to enhance understanding, we will move it to the main part in the final version.

This innovative use of image registration in our specific learning pipeline is, to our knowledge, unprecedented in both generic and medical image generation research. We want to reiterate that, image registration cannot generate counterfactual image on its own, its use in MedJourney represents an inventive adaptation of existing techniques, augmenting rather than diminishing the method's originality.