ECHOPulse: ECG Controlled Echocardio-gram Video Generation

We sincerely thank you for your thoughtful and constructive feedback, which has been invaluable in improving our work and clarifying its contributions. Your insights have provided important perspectives, and we are committed to addressing them to enhance the quality of our manuscript. We have submitted the revised manuscript, with all revisions highlighted in blue. The weaknesses you mentioned are also addressed in the Questions section. Due to space limitations, we have responded to them all within that part of the paper.

Question Response:

[Q1]Computational Cost: Regarding EchoNet-Synthetic, we did not include its inference time in the original paper because our reproduction on our private dataset showed that ECHOsynthetic takes approximately 24 seconds to generate 64 frames on a single A100 80G, likely due to differences in inference methods. Since this result deviates from their reported performance, we opted not to include it. We have included this result in the revised version of the table and details of the inference time calculation methods in the appendix.

Additionally, we feel that EchoNet-Synthetic is essentially a video editing method. Its input is essentially a duplicated image sequence that undergoes video editing with VAE decoding. Like our model, ECHOsynthetic benefits from VAE's decoding speed. However, their method generates 64 frames (three heartbeats) based on a single EF value under diffusion randomness, while our model generates 64 frames corresponding to eight complete ECG cycles, representing eight heartbeats. This significant difference in information density and prediction difficulty makes direct speed comparisons somewhat unfair.

[Q2]Typographical Error: It is really sorry that we made this mistake in Section 2.3 It should be EchoDiffusion, We sincerely apologize for the oversight caused by our typographical error. EchoDiffusion is a method based on diffusion to synthetic ECHO videos while EchoNet-Dynamic is the dataset. We have revised it in the newest version.

[Q3]Conditional Prompts: We apologize for any misunderstanding caused by our wording. When we mentioned methods requiring a large number of complex conditions, we were specifically referring to HeartBeat. In contrast, EchoDiffusion and EchoNet-Synthetic only require an image and EF value as inputs. We have revised our description to make this distinction clearer and more accurate.

That said, it is worth noting that EF is itself a complex metric. Calculating EF requires deriving ED and ES areas from the patient’s heart video, which involves a detailed and intricate process. Thus, EF can also be considered a condition obtained through a complex procedure, often required experts’ annotation, which is also the reason why their model can only work for video synthetic while ours have the potential for clinical application. Thank you for bringing this to our attention.

[Q4]Citation: We have added the correct citation. Thank you very much for correcting us.

[Q5]Comparison between VAE and Diffusion: Thank you for pointing this out. We have revised this part of the description to compare only the VAE component of EchoNet-Synthetic with our model. The results have been updated in the revised manuscripts and the reproducing details can be found in Appendix. Our intention is not to claim that our VQVAE outperforms diffusion models but rather to demonstrate that our VQVAE training was successful. The discretization process did not result in significant information loss, which validates the effectiveness of our approach. Our VQVAE did not beat the diffusion model, we only defeated the EchoNet-Synthetic’s VAE in this task. We have revised the wordings to make it more accurate. We really appreciate your feedback.

[Q6]Video Generation Comparison: Moonshot and VideoComposer directly use the results reported in the HeartBeat paper, as these two general-purpose models were not retrained. However, since the original HeartBeat paper does not provide sufficient implementation details and has not released its source code, we did not conduct comparisons on our private dataset. Instead, we supplemented our experiments with EchoDiffusion and EchoNet-Synthetic, according to your suggestions. Results can be found in the revised version.

Our private dataset includes ECG data, with each video corresponding to an ECG signal. This alignment allows the condition image-generated videos to have corresponding ground truth videos, enabling us to compute SSIM as well. Thank you for your understanding, and we hope this clarification addresses your concerns.

[W4] Inconsequent Comparison: The results of EchoNet-Synthetic in Table 1 were provided in our previous response, and a comparative summary is included below. We reproduced two methods on our private dataset, and their metrics have been included in the revised version. It is evident that our model outperforms the others. As mentioned earlier, EchoNet-Synthetic is essentially a video editing model, making direct comparisons with our generative model somewhat unfair.

On our dataset, the performance of these two methods was not as strong as reported in their original papers. This discrepancy may be due to missing details in their training processes or codebases. Even so, our model demonstrates superior performance on a more complex private dataset, while they achieve better results only on high-quality datasets.

[W5]Motivation: The theoretical foundation for using ECG as a conditioning input lies in its capacity to approximate EF to some extent. Specifically, the ratio between the first and second peaks in an ECG signal can provide a rough estimate of EF. This serves as the basis for using ECG to generate videos, and we have provide a detailed explanation of this in the Appendix. There are also works using ECG to generate reports, which also helps illustrate why we choose ECG as condition[1].

However, due to inherent inaccuracies and other factors, EF estimates derived from ECG are not as precise as those from videos. This limitation means ECG cannot serve as a standalone diagnostic tool. Our video inputs inherently contain EF-related information but also provide richer temporal details, which is why we chose ECG as the conditioning input for video generation.

[W6] Table: We have optimized the tables to improve their clarity and visual presentation. Thank you again for your constructive feedback.

Question Response:

[Q1] Results on Pre-training only on natural dataset: For this task, our experiments show that pretraining solely on ECHO videos leads to a significant decline in quality. This may be due to two primary reasons: first, the varying quality of ECHO videos makes it challenging to train a robust codebook; second, the limited quantity of ECHO video data restricts the codebook size, which in turn impacts performance.

Another important consideration for this project is the potential privacy concerns. It is difficult to guarantee that the generated videos are entirely free of patient-identifiable information. To address this, we deliberately avoided using proprietary ECHO datasets for training the VQ-VAE. Instead, we trained it on natural video datasets, which not only helps mitigate hallucination issues but also ensures high-quality video generation.

Thank you again for your suggestions; they have been incredibly helpful to us. If you have any further questions, please don't hesitate to let us know.

Reference

[1]Knight, Elizabeth, et al. "Wearable-Echo-FM: An ECG-Echo Foundation Model for Single Lead Electrocardiography." Circulation 150.Suppl_1 (2024): A4145351-A4145351.

We sincerely thank you for your thoughtful and constructive feedback, which has been invaluable in improving our work and clarifying its contributions. Your insights have provided important perspectives, and we are committed to addressing them to enhance the quality of our manuscript. We have submitted the revised manuscript, with all revisions highlighted in blue.

Weakness Response:

[W1]Quality Evaluation: Thank you for your valuable suggestions. We realize that a single figure, like Figure 3, is insufficient to fully explain our results. We have added more examples in the Appendix. We have revised the annotations and provide more detailed explanations of the relevant medical concepts in the Appendix. The intention of Figure 3 was to illustrate that ED and ES phases correctly align with the corresponding ECG phases in the generated video frames. As the ECG signal changes, the positions of ED and ES also shift accordingly. Metrics such as FID and FVD also provide supporting evidence of this alignment. Regarding the slightly lower R² compared to ECHOdiffusion, this can be attributed to the differences in training datasets. ECHOdiffusion was trained exclusively on a dataset of four-chamber heart views with relatively high-quality data and minimal EF variability. In contrast, our dataset is over nine times larger, including diverse conditions that introduce more EF variability. According to reviewer’s comets, we have reproduced EchoDiffusion and EchoNet-Synthetic on our private datasets and revised the metrics in the newest version of paper. The results shows that our model has a better performance in all aspect.

[W2]Table 3: Regarding EchoNet-Synthetic, we did not include its inference time in the original paper because our reproduction on our private dataset showed that EchoNet-Synthetic takes approximately 24 seconds to generate 64 frames on a single A100 80G, likely due to differences in inference methods. Since this result deviates from their reported performance, we opted not to include it. However, we have included this result in the revised version of the table and detail the inference time calculation methods in the appendix. Additionally, we feel that EchoNet-Synthetic is essentially a video editing method. Its input is essentially a duplicated image sequence that undergoes video editing with VAE decoding. Like our model, EchoNet-Synthetic benefits from VAE's decoding speed. However, their method generates 64 frames (three heartbeats) based on a single EF value under diffusion randomness, while our model generates 64 frames corresponding to eight complete ECG cycles, representing eight heartbeats. This significant difference in information density and prediction difficulty makes direct speed comparisons somewhat unfair.

[W3]Reconstruction influence: We are grateful for your critique and have provided the rFID and rLVEF results, which we have included in the Appendix. As noted in the paper, VQVAE discretizes variables, leading to some loss of information compared to continuous-variable models like VAE. However, according to the compression ratio concept outlined in the VQGAN paper, our selected codebook size (8192) and video patch size (8×8) yield a compression ratio of 19.69 for videos with a resolution of 128×128. Empirical experience suggests that compression ratios between 4 and 32 generally do not significantly affect video generation quality. Our parameter choices were based on these guidelines.

Our intention with Table 1 was not to claim that our method outperforms EchoNet-Synthetic’s VAE in reconstruction but rather to demonstrate that our discrete VAE achieves comparable performance to continuous VAE models, validating our video generation capability. The discrete representation was chosen to leverage the advantages of LLM-like training and prediction, at the cost of some video information loss. Table 1 sufficiently demonstrates that our method does not lose more information than existing ECHO generation models.

We have also provided EchoNet-Synthetic’s reconstruction performance on our private dataset. It is worth noting that our VQVAE was pretrained on natural images and fine-tuned on the CAMUS dataset (500 high-quality videos) rather than trained on private data. This decision was made to avoid learning potentially sensitive patient-specific features, ensuring privacy preservation. Despite this, our VQVAE still achieves commendable reconstruction performance.

In contrast, EchoNet-Synthetic’s VAE exhibited weaker generalization and required retraining on our private dataset to achieve performance similar to their original paper. This raises concerns about privacy, as their VAE was trained on private datasets despite their acknowledgment of privacy issues. If we were to train our VQVAE on private data, the reconstruction metrics would likely improve further, but privacy concerns remain a priority for us.

We sincerely thank you for your thoughtful and constructive feedback, which has been invaluable in improving our work and clarifying its contributions. Your insights have provided important perspectives, and we are committed to addressing them to enhance the quality of our manuscript. We have submitted the revised manuscript, with all revisions highlighted in blue.

Weakness Response:

[W1] Font issue: Thank you very much for bringing this to our attention. We have modified all the fonts in the figures. Moving forward, we will be even more vigilant in adhering to formatting guidelines to maintain the highest level of professionalism and accuracy in our work.

[W2] Complexity Concern: Thank you for your detailed feedback and for highlighting the need for additional parameters to complement generation speed for real-time detection. Your suggestion has helped us recognize the importance of providing more comprehensive evaluations. To address this, we will include specific details such as GPU memory usage (2.4 GB) and single-frame FLOPs (560 MFLOPs per sample). These demonstrate the feasibility of implementing our approach on wearable devices.
Our current pipeline involves collecting ECG signals using devices such as the Apple Watch or other wearables, which are then transmitted to a smartphone for video generation[1,2]. Based on previous experience, we are confident in the practicality of this approach. Your insights will help us better communicate these aspects in the manuscript.

[W3] Over-engineered Concern: Thank you for your thoughtful comments on model optimization. We greatly appreciate your recognition of the importance of simplifying the model structure while preserving its potential for future advancements. We chose VQ-VAE as the framework primarily because, in the rapidly evolving field of generative models, tokenizing ECHO videos can significantly benefit future research and applications. Given the potential impact, we believe that increasing the model's complexity for this purpose is a meaningful and justified decision. We will carefully consider your suggestions to optimize the overall model and reduce the number of modules.

The modular design in our current approach was intended to enable future scalability, allowing components to be replaced with more advanced models as they become available. For example, we aim to train an autoregressive model to replace the existing transformer-based video generation module. Similarly, the robust VQ-VAE model was trained with the goal of enhancing the framework's potential for future improvements. Your feedback encourages us to revisit these aspects and refine our explanations in the paper.

[W4] Application Scenario: We sincerely thank you for your insightful observations on the challenges associated with ECHO video training and your recognition of our efforts to address these issues. Your comment on the lack of high-quality annotated data is highly relevant, as it represents a key obstacle to advancing automated models in this domain. One of the major contributions of our work, as well as previous research, is to provide synthetic data to mitigate this challenge.

We introduced ECG signals as conditioning inputs to enhance the realism of generated videos. This is because ECG reflects cardiac muscle motion and contains temporal information, making it more suitable than other traditional conditioning inputs (e.g., EF or segmentation masks) in terms of fidelity and trustworthiness. Introducing diffusion models to the pipeline might help. While we acknowledge that diffusion models can enhance detail by imposing additional constraints, we also recognize the trade-off in terms of increased engineering complexity and inference time. Given that our current evaluation metrics (EF estimation, FID, FVD, and expert judgment) indicate satisfactory performance, we decided not to include diffusion models in this work. However, your suggestion aligns with our long-term goals, and we are actively exploring this direction for future clinical testing. Your feedback is invaluable as we refine our strategy.

[W5] Expert Evaluation: Thank you for your suggestion to include quantitative evaluations. We agree that providing quantitative results strengthens the credibility of our findings. To address this, we engaged clinicians from the hospital to perform quantitative evaluations, and we have include these results in the Appendix A.4 to further substantiate our claims. Your feedback on this matter has been extremely helpful, and we are committed to ensuring our manuscript is as robust and informative as possible.

[W6] Hallucination: Answer is the same as Q3

[W7] Downstream Task: Answer is the same as Q2

We sincerely thank you for your thoughtful and constructive feedback, which has been invaluable in improving our work and clarifying its contributions. Your insights have provided important perspectives, and we are committed to addressing them to enhance the quality of our manuscript. We have submitted the revised manuscript, with all revisions highlighted in blue.

Weakness and QuestionGre Response:

[W1] Practical Applications: Thank you for your thoughtful feedback. Your concern is something we have also considered throughout the project development phase. First, our model is capable of generating results even without a condition image. Second, patients requiring ECHO cardiac assessments typically undergo relevant tests in hospitals. Therefore, we select high-quality images with favorable angles from hospital-acquired ECHO videos as the template condition image. If patients’ data is missing, these template image may play a temporary role. Additionally, we have incorporated a LoRA module in the video tokenization stage to enable personalized customization based on a patient’s specific data. We acknowledge that the current approach is not yet perfect for clinical use, but we are actively working on improvements. Clinical applications require extensive experimentation and a lengthy validation process, and we are committed to continuously refining and enhancing our methods.

[W2] Generation Time: Thank you for your comments. We recognize that simply improving generation speed is insufficient for real-time detection; providing additional parameters is also crucial. To address this, we provide the following details: GPU memory usage is 2.4 GB, and single-frame FLOPs (per sample) is 560 MFLOPs. This makes implementation on wearable devices feasible. Our current development pipeline involves collecting ECG signals using devices such as Apple Watch or other wearables and transmitting the data to a smartphone for video generation. Furthermore, we are employing model compression techniques[1,2] to further optimize the process. Based on our previous experience, this approach is viable and has the potential to achieve real-time functionality.

Reference:

[1] Niu, Wei, et al. "SmartMem: Layout Transformation Elimination and Adaptation for Efficient DNN Execution on Mobile." Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 3. 2024.

[2]Shu, Zhihao, et al. "Real-time Core-Periphery Guided ViT with Smart Data Layout Selection on Mobile Devices." The Thirty-eighth Annual Conference on Neural Information Processing Systems.

We sincerely thank you for your thoughtful and constructive feedback, which has been invaluable in improving our work and clarifying its contributions. Your insights have provided important perspectives, and we are committed to addressing them to enhance the quality of our manuscript. We have submitted the revised manuscript, with all revisions highlighted in blue.

Weakness Response:

[W1] Qualitative Comparison: Thank you for your valuable suggestions. According to your suggestion, we have added the comparison of reconstruction results with EchoNet-Synthetic in Table 1. And we have added Figure 11 in Appendix to show the reader the visualization results of model’s reconstruction performance.

[W2]Lack of Video Generation Comparison: Thank you for your insightful feedback. We have added additional figures in the appendix, which compares our method with current models to further emphasize our model’s privilege.

[W3] Add Diffusion Model to Pipeline: We appreciate your suggestion. Other reviewers have expressed similar concerns. We have indeed considered incorporating diffusion models into the pipeline to impose stronger constraints on video generation and improve video quality. However, we are somewhat concerned that this may overly complicate the video generation process. Moreover, the current results, both in terms of quantitative metrics and expert evaluations, do not reveal significant issues. Nevertheless, we are actively exploring and researching this direction.

Questions Response:

[Q1]Qualitative Comparison: Thank you for your comments. While our tokenization model was designed and trained with significant effort, it is not the primary contribution we aim to highlight in this paper. The comparisons with other ECHO video generation methods are intended to demonstrate that our video tokenization model is effective and performs on par with established approaches. The success of this paper does not primarily rely on the performance of the video tokenization model. Other video generation models based on VQ-VAE, such as Phenaki[1], MAGVIT[2], and MAGVIT2[3], are not open-sourced, making it impossible for us to compare with them.

[Q2]Exploration of Over-lapping Patches: Regarding the suggestion about overlapping patches, we have indeed experimented with similar approaches. However, for this task, the performance improvement was not significant, and we did not observe artifacts in the current results. Additionally, given that the VQ-VAE codebook requires searching, overlapping patches would increase the computational burden. For these reasons, we decided not to further pursue this direction at the time.

Thank you again for your suggestions; they have been incredibly helpful to us. If you have any further questions, please don't hesitate to let us know.

Reference: [1] Villegas, Ruben, et al. "Phenaki: Variable length video generation from open domain textual descriptions." International Conference on Learning Representations. 2022. [2] Yu, Lijun, et al. "Magvit: Masked generative video transformer." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023. [3] Yu, Lijun, et al. "Language Model Beats Diffusion--Tokenizer is Key to Visual Generation." arXiv preprint arXiv:2310.05737 (2023).