SeLoRA: Self-Expanding Low-Rank Adaptation of Latent Diffusion Model for Medical Image Synthesis

We appreciate Reviewer Pa5H’s valuable and constructive feedback regarding the analysis of computational complexity and the limited modalities in our experiments. In response to these concerns, we have conducted new analyses and additional experiments. Below, we address each point raised in detail:

Including Other Modalities

To address this, we have included an additional experiment on pathology images - the PatchGastricADC22 dataset, demonstrating the robustness and versatility of our method. The results are shown in Table 6 and are also showcased below:

On the PatchGastricADC22 dataset, while our method ranked second, it is comparable to the best-performing method (LoRA) and only 0.5 points behind. Additionally, our method exhibited the lowest training time and GPU memory usage on PatchGastricADC22 as shown below, justifying the slight degradation in FID. This demonstrates a favorable trade-off between performance and training efficiency. Furthermore, we plan to validate our method on additional modalities, such as MRI, CT, or ultrasound datasets, in future work.

Training Efficiency Analysis

To facilitate a training efficiency analysis, we conducted experiments for SeLoRA and related methods using the IU X-ray dataset, as well as the PatchGastricADC22 dataset. During these experiments, we recorded the training time and GPU memory usage at intervals of every 40 training steps. The results are presented in Table 8 (training time) and Table 7, Figure 11, and Figure 12 (GPU memory usage). For ease of viewing, the training time and average GPU memory usage are summarized below:

For training time, our method achieved the shortest training time on the PatchGastricADC22 dataset and ranked third for the IU X-Ray dataset. However, on the IU X-ray dataset, our method was only 6 minutes slower than the fastest method (DyLoRA) — a negligible difference. When analyzing training time in conjunction with the FID evaluation in Tables 5 and 6, we observe a significant advantage. On the IU X-Ray dataset, our method outperforms the fastest method (DyLoRA) by 23 points in FID. Meanwhile, on the PatchGastricADC22 dataset, where our method is the fastest, it incurs only a minor 0.5-point disadvantage in FID compared to the best-performing method (LoRA). This demonstrates that our method offers a highly advantageous trade-off between training time and performance.

For GPU memory usage displayed in Figures 11 and 12, it is evident that our method never required the highest GPU memory among the compared methods and consistently had the lowest memory usage. In Table 7, which reports the average GPU memory usage, our method had the lowest memory usage on the PatchGastricADC22 dataset and ranked second on the IU X-Ray dataset. Similar to the training time results, although our method is slightly behind DyLoRA on the IU X-Ray dataset in GPU memory usage, the significant 23-point advantage in FID justifies this trade-off.

Therefore, we believe our method strikes the best balance between performance and training efficiency.

We appreciate Reviewer 7CbA's thorough feedback, particularly regarding the analysis of computational complexity and our evaluation. We have strengthened our work by conducting additional experiments and analyses to address these concerns. Our detailed responses follow:

Computational Complexity Analysis

To facilitate computational complexity analysis, we conducted experiments for SeLoRA and related methods using the IU X-ray dataset, as well as an additional PatchGastricADC22 dataset. During these experiments, we recorded the training time and GPU memory usage at intervals of every 40 training steps. The results are presented in Table 8 (training time) and in Table 7, Figure 11, and Figure 12 (GPU memory usage). For ease of viewing, the training time and average GPU memory usage are summarized below:

Key results:

• SeLoRA achieves the best FID for both datasets.
• SeLoRA consistently balances training time, memory usage, and performance.
• The additional overhead of Fisher Information computation is minimal, making SeLoRA efficient even in resource-constrained settings.

For the training time, our method has the shortest training time on the PatchGastricADC22 dataset and ranks third for the IU X-ray dataset. However, on the IU X-ray dataset, our method is only slightly behind the fastest method (DyLoRA) by just 6 minutes—a negligible difference. When analyzing training time in conjunction with the FID evaluation in Tables 5 and 6, we observe a significant advantage. On the IU X-ray dataset, our method outperforms the fastest method (DyLoRA) by 23 points in FID. Meanwhile, on the PatchGastricADC22 dataset, where our method is the fastest, we incur only a minor 0.5-point disadvantage in FID compared to the best-performing method (LoRA). We believe this suggests that our method offers a highly advantageous trade-off between training time and performance.

For GPU memory usage displayed in Figures 11 and 12, it is evident that our method never required the highest GPU memory among the compared methods and consistently used the lowest memory. In Table 7, which reports the average GPU memory usage, our method had the lowest memory usage on the PatchGastricADC22 dataset and ranked second on the IU X-ray dataset. Similar to the training time results, although our method is slightly behind DyLoRA on the IU X-ray dataset in GPU memory usage, the significant 23-point advantage in FID makes this trade-off justifiable.

Therefore, we believe our method strikes the best balance between performance and training efficiency.

Computation Overhead for Fisher Information

Regarding the computational overhead for calculating Fisher information, it can be efficiently computed by taking the gradient and performing element-wise squaring of each matrix element. Therefore, the primary computational overhead comes from computing the model's gradient through backpropagation, while the additional overhead of squaring and summing the elements is negligible. If we estimate the additional computational complexity introduced by calculating Fisher information, it will be around $\frac{1}{t}$, where $t$ refers to the rank testing interval. In our case, with $t = 40$, this translates to an additional 2.5% overhead in terms of computational cost. Given this, we believe that the computation of Fisher information introduces a modest and manageable overhead, without significantly impacting the overall training efficiency.

Evaluation of New Modality and Larger Datasets

To address this, we have included a new experiment on pathology images using the PatchGastricADC22 dataset to demonstrate the robustness and versatility of our method. The results are shown in Table 6. On the PatchGastricADC22 dataset, while our method ranked second, it was only 0.5 points behind the best-performing method (LoRA). Additionally, our method exhibited the lowest training time and GPU memory usage on PatchGastricADC22, justifying the slight increase in FID. This demonstrates a favorable trade-off between performance and training efficiency. Furthermore, we plan to validate our method on additional modalities, such as MRI, CT, or ultrasound datasets, in future work.

Evaluating SeLoRA on larger datasets, such as MIMIC-CXR (which contains approximately 377,000 images), would provide further insights into its scalability. However, the primary challenge that medical image synthesis aims to address is the scarcity of labeled medical images. This is precisely why fine-tuning, rather than training from scratch, becomes crucial in such contexts. If a dataset of the scale of 377,000 images (comparable to ImageNet-1K) were available, there would be less incentive to generate additional synthetic data. In such cases, the focus would shift towards directly training models on the original dataset to leverage the abundant real-world data.

As a reference for the size of datasets typically used to fine-tune LoRA on Stable Diffusion (SD), a prior ICLR paper on LoRA adaptation for SD utilized around 45–200 images per class, totaling approximately 1,706 images, as noted in Yeh et al., 2024. In contrast, MIMIC-CXR is significantly larger, with 40,000 images per class. We believe that fine-tuning on smaller datasets is a more appropriate and practical use case for LoRA-based methods, particularly in scenarios like ours, compared to training on the full 377,000-image MIMIC-CXR dataset.

Incorporating Related Work

We acknowledge that the idea of dynamically adjusting the rank of the LoRA matrix in SeLoRA shares conceptual similarities with the recently proposed ALoRA which has been tested language models, and we have included it in our related work section. Indeed, ALoRA could provide valuable inspiration. This caused our immediate interests in the two types of philosophy behind ALoRA and SeLoRA. In future work, we plan to explore the integration of pruning mechanisms alongside our Fisher information-guided rank adjustment to further enhance SeLoRA’s flexibility and efficiency. It would be especially interesting to compare ALoRA's efficiency, where it reach its peak of GPU memory usage at the beginning of training, aganist SeLoRA, where the the maximum of GPU memory usage arrived at the end.

Medical Doctor Evaluation

We have engaged medical doctors for evaluations that are currently underway. However, we do not anticipate completing these evaluations before the rebuttal deadline. We plan to include these expert assessments in future work to thoroughly evaluate the model's ability to generate clinically relevant images based on disease or abnormal findings. In the meantime, to address this concern partially, we have conducted additional experiments on the IU X-ray dataset with higher resolution, as well as on another modality using the PatchGastricADC22 dataset. These experiments demonstrate the robustness of our model.

Data Split of the IU X-Ray Dataset

To address this concern, we have repartitioned the IU X-Ray dataset into a 70%-10%-20% split for training, validation, and testing, respectively, to ensure a more robust evaluation with a larger test set. The results, presented in Table 5 in the updated manuscript, show that our method continues to achieve the best FID among all LoRA-based methods.

We thank Reviewer Ko6A for the valuable and constructive feedback on the data size, limited modality, and the suggestion to explore higher image resolution. In response, we have conducted additional analyses and experiments to address these concerns. Below, we provide a point-by-point response:

Visual Quality Compared to Chambon et al. (2022a;b)

We acknowledge the difference in visual quality. Chambon et al. fine-tuned their model with 175k images using 64 A100 GPUs, whereas our method was fine-tuned with only 2k images on a single T4 GPU. Despite these disparities, SeLoRA stands out by prioritizing efficiency and adaptability in resource-constrained environments and low-data regimes, setting it apart from prior work. Additionally, we have provided further results at a higher resolution (384×384), highlighting improved visual quality and its potential for broader evaluations.

Size of Datasets

Evaluating SeLoRA on larger datasets, such as MIMIC-CXR, which contains approximately 377,000 images, could offer deeper insights into its scalability. However, the core challenge in medical image synthesis is the scarcity of labeled medical images. This underscores the importance of fine-tuning over training from scratch in such scenarios. If a large dataset like MIMIC-CXR (comparable to ImageNet-1K) were available, the need for generating synthetic data would diminish. In such cases, the focus would shift to directly training models on the original dataset to leverage abundant real-world data.

As a reference, prior ICLR papers on LoRA adaptation for Stable Diffusion (SD), such as Yeh et al. (2024), utilized datasets containing approximately 45–200 images per class, totaling around 1,706 images. In contrast, MIMIC-CXR is significantly larger, with approximately 40,000 images per class. We believe fine-tuning on smaller datasets is a more appropriate and practical use of LoRA-based methods, particularly in scenarios like ours, compared to training on the full 377,000-image MIMIC-CXR dataset.

To address concerns about small test sets, we conducted experiments using a repartitioned dataset with a 70%-10%-20% split for training, validation, and testing. The results, presented in Tables 5 and 6, show that our methods achieve the best FID for all dimensions on the IU X-Ray dataset.

Clarifications on Minors

Dataset split. We have conducted an experiment on a repartitioned dataset with a 70%-10%-20% split for training, validation, and testing.

Table 1 was computed using the test dataset.

SeLoRA's advantage. The Montgomery County CXR dataset contains only about 100 training samples, compared to the larger IU X-Ray dataset used by other methods. In this low-data regime, most LoRA-based methods failed to learn meaningful representations and instead produced noisy or unrealistic lung images. This limitation stems from their uniform-rank or pruning designs, which hinder effective learning with small datasets.

In contrast, our method dynamically allocates rank to the most important layers, guided by Fisher information. By adapting rank placement, SeLoRA captures the critical features necessary for generating high-quality images, even with limited data. This capability highlights SeLoRA's distinct advantage in low-data settings, a strength that is especially vital in the medical imaging domain.

More evaluation. Since there are no readily available chest image classification models specifically trained on the IU X-Ray dataset, we have engaged medical experts for evaluations. However, we do not anticipate having sufficient time to complete these evaluations before the rebuttal deadline. We plan to include these expert assessments in future work to provide a more thorough evaluation of the model’s ability to generate clinically relevant images. In the meantime, to partially address this concern, we conducted additional experiments on IU X-ray with higher resolution and on a new pathology dataset as below.

Additional Experiments on New Dataset and Higher Resolution

New dataset. We acknowledge the reviewer’s observation regarding the focus of our work on chest X-ray images, given the title’s broader scope of "medical image synthesis". To address this, we have included an additional experiment on the PatchGastricADC22 dataset, representing a different modality and highlighting the robustness and versatility of our method. The results are shown in Table 6 and summarized below:

On the PatchGastricADC22 dataset, while our method ranked second, it was only 0.5 points behind the best-performing method (LoRA). Additionally, our method exhibited the lowest training time and GPU memory usage on PatchGastricADC22, justifying the slight increase in FID. This provides a good trade-off between performance and training efficiency.

Higher resolution. We have conducted additional experiments on the PatchGastricADC22 and IU X-Ray datasets with a higher resolution of 384x384. The FID scores are reported in Tables 5 and 6. For convenience, we summarize the results below:

For the IU X-ray dataset, our method achieved the best FID across all dimensions, demonstrating its superiority. On the PatchGastricADC22 dataset, while our method ranked second, it was only 0.5 points behind the best-performing method (LoRA) Additionally, our method exhibited the lowest training time and GPU memory usage on PatchGastricADC22, justifying the slight increase in FID. These results show that our method provides a good trade-off between performance and training efficiency. The new experiments, conducted with a resolution of 384x384, further convey the robustness of our method.

We appreciate Reviewer ob8y’s valuable and constructive feedback on the analysis of training efficiency and hyper-parameter selection. We have conducted new analyses and additional experiments to address the concerns. Below we address your raised issues point-by-point. Our manuscript is updated accordingly.

Training Efficiency Analysis

To analyze training efficiency, we conducted experiments for SeLoRA and related methods on the IU X-ray and PatchGastricADC22 datasets, recording training time and GPU memory usage every 40 steps. The results are detailed in Table 8 (training time) and Table 7, Figures 11 and 12 (GPU memory usage) in the updated manuscript. Key findings are summarized below:

---

---

Key results:

• SeLoRA achieves the best FID for both datasets.
• SeLoRA consistently balances training time, memory usage, and performance.
• The additional overhead of Fisher Information computation is minimal, making SeLoRA efficient even in resource-constrained settings.

Our method achieves the shortest training time on the PatchGastricADC22 dataset and ranks third on the IU X-Ray dataset, trailing the fastest method (DyLoRA) by only 6 minutes—a negligible difference. When combined with the FID evaluation (Tables 5 and 6), the advantage becomes clear: on the IU X-Ray dataset, our method outperforms DyLoRA by 23 FID points. On the PatchGastricADC22 dataset, where our method is the fastest, it incurs only a minor 0.5-point FID disadvantage compared to the best-performing method (LoRA). These results highlight a favorable trade-off between training time and performance.

For GPU memory usage (Figures 11 and 12), our method consistently ranks among the most efficient. It achieves the lowest memory usage on the PatchGastricADC22 dataset and ranks second on the IU X-Ray dataset. While slightly behind DyLoRA in memory efficiency on the IU X-Ray dataset, the advantage of 23 points in FID justifies this trade-off.

Overall, our method strikes the best balance between performance and training efficiency.

Hyper-parameter selection

We appreciate the reviewer’s concerns about hyperparameter selection and provide the following clarifications:

• $\lambda$ Parameter: $\lambda$ represents the ratio between the previous and current Fisher Information (FI) scores, typically close to 1. Our analysis shows $\lambda$ = 1.1 is optimal for capturing incremental improvements, as detailed in Section 4.2 (Ablation Study). Even with $\lambda$ = 1.3, performance degradation is minimal (a 9-point FID increase), demonstrating the method's robustness and flexibility.

• $t$ Parameter: A larger $t$ ensures reliable rank expansion by evaluating Fisher Information over a meaningful batch size. This prevents unnecessary rank increases and maintains training efficiency.

These clarifications, supported by ablation results, highlight the robustness of our approach to variations in these parameters.

Clarification on Noisy Images in Figure 7

Notably, the Montgomery County CXR dataset has only ~100 training samples. Most LoRA-based methods failed due to uniform rank allocation, resulting in noisy outputs. SeLoRA’s adaptive rank allocation enabled it to generate meaningful images even in this low-data regime, a critical advantage for medical imaging applications.

Comparison to Adaptive Rank Selection Methods

Adaptive rank selection methods, such as Generalized LoRA, rely on computationally expensive evolutionary searches to determine optimal layer-wise rank configurations. Their process involves training ~50 subnets from a supernet, followed by evolutionary search and additional training of selected subnets, significantly increasing training time. As noted in their paper, “training time may increase due to this search process,” and comparisons are often made against hand-tuned hyperparameter configurations for LoRA.

In contrast, SeLoRA and the methods we benchmarked against (DyLoRA and AdaLoRA) determine the optimal rank dynamically during a single training run, significantly reducing training time. While adaptive rank selection can yield strong results with extensive computational resources, its design is impractical for resource-constrained environments.

Comparing SeLoRA with adaptive rank selection methods would be inherently unfair due to the stark disparity in training time and computational costs. Instead, our work focuses on achieving competitive results within a single training process, aligning with the practical constraints of real-world applications.