Selective LoRA for Domain-Aligned Dataset Generation in Urban-Scene Segmentation

[Q4.1] Enhancing Clarity and Presentation for Better Understanding

We deeply appreciate the reviewer for recognizing the technical novelty and strengths of our experiments. Acknowledging the importance of improving the presentation of our paper, we have made the following revisions in Sections 3 and 4:

• We revised Figure 1 to highlight the issues found in previous studies. DatasetDM does not involve fine-tuning the pretrained T2I model on the source dataset, and we identified the overfitting problem when training all layers in the Original LoRA.
• In the method section, we updated Figure 3 to help understanding of Section 3.2.
• The explanation of the segmentation dataset generation framework was insufficient. We added detailed explanations in Section 3.1 Overall Framework and Section 3.4 Training Label Generator and Generating Diverse Segmentation Datasets.
• We reorganized the entire Section 4. Experiments section.

Thank you for your constructive feedback. We have carefully addressed your concerns below, and the proposed changes have been fully integrated into the revised manuscript. We welcome any additional input you may have.

[W4.1] I believe that the proposed method tackles an important problem and offers a reasonable approach to addressing the challenges, which I highly commend. However, there are some weaknesses to consider.

Thank you for acknowledging the significance of our proposed method and its approach to addressing the challenges. We sincerely appreciate your commendation. Regarding the weaknesses mentioned, we have carefully addressed them in the revised manuscript and provided additional clarifications and experiments where necessary. We believe these revisions strengthen the paper and hope they adequately address your concerns. Please let us know if there are any further aspects that require clarification or improvement.

[W4.2] Limitation of Reliance on Pretrained Stable Diffusion Models

We agree with the reviewer’s observation that a key limitation of our work is its reliance on the knowledge embedded in the pretrained Stable Diffusion model. While the additional information is indeed constrained by the prior knowledge of the pretrained text-to-image generation model, we have demonstrated significant improvements in urban-scene segmentation across both in-domain and domain generalization tasks. Furthermore, we believe that the proposed segmentation dataset generation framework has the potential to harness the extensive prior knowledge of large-scale text-to-image generation models for semantic segmentation by employing a selective adaptation methodology. This aspect has been addressed in the revised manuscript in Section 5: Conclusion and Future Work, L530–534.

[W4.3] Challenges in Manual Concept Definition and Model Adaptation

As the reviewer pointed out, the process of defining desired concepts can indeed be quite manual. However, our main argument is that in cases where the desired concept is clear, such as urban-scene segmentation, our methodology provides an effective approach to exclusively learn the desired concept. Extending this approach to identify and adapt desired concepts for other target datasets, as we have demonstrated for urban-scene segmentation, could be an exciting topic for future research.

Furthermore, the following newly conducted experiments demonstrate the robustness of our method:

[Pascal VOC] For in-domain tasks, even beyond urban-scene datasets, training on Pascal-VOC using the identified style-sensitive layers effectively captured the style of Pascal-VOC, resulting in additional performance improvements. These results are detailed in Figure 13 and Table 9 in Appendix A.8: In-Domain Experiments for the General Domain Dataset (Pascal-VOC).

[Robustness to Prompt Augmentation] As shown in Figure 12 in Appendix A.7: Concept Sensitivity According to Prompt Augmentation, we found that when the desired concept (style or viewpoint) is well-defined, the sensitivity score remains consistent across various prompt augmentations, highlighting the robustness of our approach.

Additionally, finding the associated weights and determining their importance requires experimental work, which lacks standardized criteria.

As the reviewer noted, determining the proportion of selected layers may involve experimental work. However, we have introduced metrics for evaluating the quality of the generated dataset, including CMMD, CLIP-Score, and Image-Label Alignment (mIoU), as presented in Table 3, Table 4, and Table 8, which may serve as reasonable criteria. Additionally, we believe that the proportion of selected layers can be treated as a hyperparameter to adjust the exclusiveness of the desired concept relative to other concepts. We have also provided qualitative results and a detailed analysis in Figure 16, Appendix A.10: Additional Qualitative Results, and L1241-L1273, respectively.

Thank you for your valuable suggestions. We present the following additional experiments, analyses, and discussions to address each of your considerations.

1. Class-specific Performance Analysis

As the reviewer pointed out, measuring and analyzing class-wise IoU is a crucial aspect of urban-scene semantic segmentation. We have included the related experiments in Appendix A.10. Class-wise Segmentation Performance Analysis, with the key results summarized as follows.

Class-wise IoU for In-domain few-shot experiments (Cityscapes, 0.3%)

We demonstrated particularly significant performance improvements for the aforementioned classes in Figure 16. As evident from the experimental results, as you mentioned, rare classes that appear in less than 1% of the dataset showed substantial performance gains. This serves as an additional strength of our approach.

In addition, we observed that certain classes were generated at proportions lower than their actual occurrence in the Cityscapes dataset. To address this, we proposed a methodology to generate additional samples for these underrepresented classes. As shown in Figure 17, unlike the Original LoRA, the proposed Selective LoRA methodology effectively generates additional samples for specific classes, as it exclusively learns the style of the source dataset.

We conducted an experiment aimed at enhancing the performance of a specific target class (e.g., "person") by generating additional samples for that class. The result of the experiment is presented in Figure 16, which demonstrates that the additional dataset achieves more balanced performance across classes by adjusting the label proportions. The following highlights the key results of the performance improvements.

Please refer to Segmentation Dataset Generation Focused on a Specific Class in line 1283 for further details.

2. Text Guidance Considerations: Given that this is a text-guided approach, we might expect varying degrees of performance improvement based on different text variations. Have you considered using alternative class names beyond the provided prompts? How might this affect performance?

Applying text augmentations to the given class names to create more diverse text prompts for dataset generation is a highly promising research direction for our method.
To explore this, we conducted experiments detailed in Appendix A.11 Generating Datasets with Diverse Class Names, where we replaced the simple use of "Car" for generating cars with more detailed class names. The key findings are summarized as follows.

First, as shown in Figure 18, we augmented the text prompts by diversifying "Car" into more specific categories such as "SUV car", "Sedan car", "Convertible car", and "Hatchback car" for data generation. Our Selective LoRA demonstrated its ability to generate these diverse classes effectively, as it avoids overfitting to the Cityscapes content, maintaining its generalization capability.

We then incorporated these generated samples into the dataset and conducted training with the augmented dataset, as shown in Table 10.

The generated dataset enhanced overall segmentation accuracy and demonstrated consistent performance improvements in vehicle classes such as car, bus, and motorcycle. Further exploration of additional test prompts presents a promising direction for future research.

3. Domain Generalization Question: I notice that the performance gains in domain generalization scenarios are lower compared to the improvements seen in Cityscapes. Could you explain the potential reasons for this discrepancy?

Thank you for recognizing the performance improvements in the in-domain setting. However, we believe the gains in the domain generalization (DG) setting are equally noteworthy.

The effectiveness of our approach becomes even more evident when examining individual datasets rather than the average DG performance across four datasets.
We observed significant improvements on datasets with adverse weather conditions. For example, our method enhanced the performance of ColorAug by 2.95 on ACDC and 4.06 on Dark Zurich, which represent substantial gains compared to existing baselines.

Furthermore, on HRDA, one of the most advanced DG methods, the second-best baseline achieves a performance increase of 0.38 (with DGInStyle reporting only a slight improvement of 0.71), whereas our approach achieves a significantly higher improvement of 1.53.

For the potential reasons why the performance improvements may appear modest, we would like to highlight the extensive use of color augmentations already employed by existing DG methods. As noted in lines 454-460, methods such as DAFormer and HRDA incorporate aggressive color augmentations. This extreme augmentation likely diminishes the impact of dataset generation approaches, including ours, by addressing color-related variations upfront.

Nevertheless, while image-to-image translation-based methods such as DGInStyle and InstructPix2Pix show increasingly marginal gains on DAFormer and HRDA, our method, though slightly reduced, continues to deliver substantial improvements. We believe this underscores the robustness and effectiveness of our approach in addressing domain generalization challenges.

I have an opinion about the text guidance and the performance degradation or minor improvement of the class "person". As the authors used, it would be helpful to consider specific and diverse words like "SUV" and "Sedan". However, I think it will be better for authors to consider more text guidacne for the class "person".

The word "person" is fairly appropriate word to represent the class, because it represents human without any criteria. But it is too neutral for the distribution of the diffusion models' training data. Also, it is too ambiguous because of the class "rider" (in case of Cityscapes). Since "person" contains "rider", the class "person" needs to be changed into more specific words or clauses. It would be helpful to consider the word "pedestrian" which does not overlap with "rider".

However, this is also another research direction. The authors don't have to consider this opinion. This will not affect my rating.

Thank you for your valuable feedback. We have addressed your concerns in detail below, and these revisions have been reflected in the updated manuscript. If there are any additional points you'd like us to consider, please let us know.

[W3.1. W3.2] Clarification of "Informative Samples" Definition and Figure 1

We acknowledge the lack of clarity regarding "informative samples". In response to the reviewer's recommendation, we revised Figure 1 and its caption, adding our informative examples to clearly illustrate the concept. Additionally, we updated L78-L80 in the revised version to explicitly define 1) domain-aligned and 2) informative samples using the example provided in Figure 1. Below is a brief clarification of the informative sample.

Informative samples refer to generated image-label pairs that provide additional information beyond what is available in the existing source dataset, as described in L43–44. In this study, the additional information is provided by pretrained text-to-image generation models (e.g., Stable Diffusion). The definition of "informative samples" varies depending on the problem setting in urban-scene segmentation. For example, when the target domain is ACDC, which includes driving-scene viewpoints and adverse weather, but the training dataset is Cityscapes, which consists of driving-scene viewpoints with only clear-day conditions, a diverse weather conditional dataset (informative) combined with a driving-viewpoint (domain-aligned) could serve as the optimal dataset for the problem.

As illustrated in the revised Figure 1, learning and fixing the style of the source dataset (e.g., the clear-day style from Cityscapes) does not add additional information, making it uninformative for this scenario. However, it may still be considered informative in the context of in-domain segmentation. Therefore, we propose a flexible method that selectively learns only the viewpoint or style from the Cityscapes training data while avoiding overfitting to the other concepts.

[W3.3] Comparative Analysis with Image-Driven Diffusion Models (InstructPix2Pix)

Comparing our method to the suggested baseline, InstructPix2Pix, is a critical experiment. We have conducted additional comparisons with InstructPix2Pix and included the results in Table 2, Section 4.2, of our revised manuscript.

InstructPix2Pix is an image-to-image translation model that generates diverse styles with a given image, similar to DGInstyle, which was included as a baseline method in our initial manuscript. Consequently, the segmentation maps remain consistent across various generated images derived from a single image. Since InstructPix2Pix is designed to generate diverse styles of a given image, we concluded that it is not suitable for dataset augmentation in in-domain scenarios. To this end, we compared our proposed method with InstructPix2Pix using the domain generalization task using the domains of foggy, rainy, night-time, and snowy images in Cityscapes. The results are as shown.

The table demonstrates that our proposed method consistently outperforms InstructPix2Pix across all DG methods. Specifically, we found that InstructPix2Pix is less effective for recent state-of-the-art baselines such as DAFormer or HRDA. This is because our method generates diverse scenes and their corresponding segmentation maps, whereas InstructPix2Pix merely changes the styles of a given image without creating diverse scenes. Such augmentation effects are already provided by severe data augmentation methods. This experiment clearly highlights the necessity of our proposed method, given the limitations of using current image-to-image translation models as data generators.

Thank you for your efforts and the detailed response. The majority of my concerns have been addressed, but a few questions remain.

Thank you for acknowledging our efforts and detailed responses. We are pleased to hear that the majority of your concerns have been addressed. Below, we provide responses to the remaining questions, including additional experiments and analyses.

Clarification and Highlighting the Novelty of Our Contribution

Our contribution does not lie in the framework itself. As outlined in Section 2.2. Segmentation Dataset Generation, the use of generative models to create image-label pairs for enhancing perception models, has already been explored in prior works such as DatasetDM and DatasetGAN.

What we aim to highlight, as summarized in L099–L107, is our proposal of the Selective LoRA fine-tuning method, which sets us apart from DatasetDM, as it directly uses pre-trained text-to-image generation models without any fine-tuning. Our method enables selective learning of desired concepts (e.g., style, viewpoint), as demonstrated in the updated Figure 2, Stage 1 and 2. We would like to emphasize that these two aspects represent reasonable and novel contributions unique to our work, as agreed upon by all reviewers (B65c, hMNt, KMEN, kFPj).

Comparison with DATUM [6]

As per your suggestion, we conducted a comparison with DATUM [5] in the domain generalization setting. The following outlines the performance comparison between DATUM and our proposed method, which has been updated in Table 2.

DATUM [5] proposes a One-shot UDA approach. One-shot UDA refers to a setting where a real image from each target domain ("foggy", "night-time", "rainy", and "snowy") is available for domain adaptation. In contrast, domain generalization assumes no access to any real images from the target domain. DATUM leverages a single real target domain image and uses a text-to-image generation model to create a diverse set of unlabeled images for the target domain. These unlabeled image sets are then used in combination with the given source domain by leveraging existing domain adaptation techniques (e.g., DAFormer, HRDA).

On the other hand, as illustrated in Figure 2. Stage 3, our approach trains a label generator to create a labeled dataset, which is then directly mixed into the training set for further training. This fundamental difference allows our method to achieve significantly higher performance than DATUM, even though DATUM utilizes a real target domain image.

Thank you for suggesting this comparison, as it provides an excellent opportunity to highlight the additional advantages of our approach.

Comparison with PTDiffSeg [5]

We appreciate your comments and would like to address the clarification regarding the extra experiment involving PTDiffSeg.

First, PTDiffSeg and our method have different research purposes. PTDiffSeg is a segmentation model designed to resolve domain generalization (DG) tasks, not an image generation method. In contrast, our goal is to develop a data generation method for the segmentation field that can produce domain-aligned and diverse image-label pairs. Specifically, PTDiffSeg introduces a novel architecture by utilizing a pretrained diffusion model as its backbone and compares it with other DG methods such as ColorAug, DAFormer, and HRDA. However, our method aims to resolve the lack of datasets for any existing segmentation model, whether in-domain or DG, to improve the segmentation performance. For example, our method can be applied to PTDiffSeg to achieve further improvements, similar to its application in DAFormer or HRDA. However, it can not offer a direct comparison between PTDiffSeg and our proposed approach.

Second, unfortunately, PTDiffSeg's code is not currently publicly available, which causes challenges in conducting a quick comparison experiment. We are currently struggling with re-implementation. It is an intriguing experiment to investigate whether our generated dataset can effectively complement an advanced DG method like PTDiffSeg. However, we have concerns due to the re-implementation issue and the rebuttal schedule. Therefore, there is a possibility that we will include these results in our paper after the rebuttal period ends.

Two Key Analysis for the Segmentation Dataset Generation: (1) domain-invariance of the generated images relative to the target domain (Image Domain Alignment) and (2) the quality of the corresponding pseudo labels (Image-Label Alignment)

We have already conducted both analyses in the revised manuscript for the in-domain setting. First, image domain alignment is discussed in the revised paper under Section 4.3. Analysis, specifically in the subsection titled Image Domain Alignment. As shown in Table 3, we provided quantitative results by measuring image domain alignment between the source dataset (Cityscapes) and the generated dataset using CMMD. Additionally, based on feedback from various reviewers, we analyzed image-label alignment qualitatively in Figure 6 and quantitatively in Table 8. More detailed analyses can be found in Appendix A.6 Comparison of Image-Label Alignment.

Thus, we interpreted the current reviewer’s question as requesting additional analyses in the domain generalization setting. To address this, we have included a new analysis in Appendix A.12. Additional Analysis of Our Generated Dataset on the Domain Generalization Setting, with the key results summarized as follows.

Image Domain Alignment in Domain Generalization Setting: (1) domain-invariance of the generated images relative to the target domain

For domain generalization, we generated additional datasets for "foggy", "night-time", "rainy", and "snowy" conditions and incorporated them into the training process. To evaluate text adherence for each condition, we analyzed the datasets using CLIP scores, as shown in Table 4.

In this additional analysis, we further investigate how well each generated image set aligns with the actual images from ACDC corresponding to "foggy", "night-time", "rainy", and "snowy" conditions. This evaluation is performed using CMMD to measure image domain alignment (newly added in Table 11).

† For DATUM, we provide an additional image for each weather condition to satisfy the requirements of the One-shot UDA setting, whereas the other methods do not rely on target domain images.

As demonstrated in the results, our generated dataset achieves significantly better image domain alignment for each adverse weather condition compared to DatasetDM and InstructPix2Pix. This improvement likely arises from our method, which exclusively learns viewpoint information from Cityscapes, effectively utilizing driving scene knowledge. By contrast, DATUM requires a single real ACDC image for each weather condition and trains four separate models, one for each condition, using the additional target domain images. Even without access to any real ACDC images and using a single model to generate datasets for multiple weather conditions, our approach achieves a comparable level of image domain alignment to that of DATUM.

Additionally, while our method currently addresses in-domain and domain generalization settings, exploring its application in a One-shot UDA setting, where additional training is performed using a single provided target domain image, would be an intriguing direction for future work.

Image-Label Alignment in Domain Generalization Setting: (2) the quality of the corresponding pseudo labels.

In this analysis, we aim to evaluate how reliably our method provides pseudo labels compared to other methods in the domain generalization setting, both qualitatively and quantitatively. The detailed analysis is provided in Appendix A.12, featuring Figure 19, Table 12, and Table 13.

First, we added qualitative results in Figure 19, which show that our method provides better labels than DatasetDM or InstructPix2Pix. For quantitative results, we utilized pretrained segmentors, as done in the analysis presented in Table 8. However, the previously used pretrained Mask2Former, trained only on Cityscapes, does not deliver reliable performance under adverse weather conditions.

To address this, we fine-tuned the pretrained Mask2Former (M2F) on the ACDC training set for each weather condition, creating specialized segmentors for "foggy", "night-time", "rainy", and "snowy". The performance of these segmentors on the ACDC validation set before and after fine-tuning is presented below.

Each fine-tuned model is now expected to provide more meaningful segmentation maps for its respective weather condition. The results of measuring Image-Label Alignment using these models are as follows.

As shown in the table, InstructPix2Pix, which directly uses Cityscapes labels and only slightly edits the weather conditions of the images, demonstrates an advantage in image-label alignment. Despite its high image-label alignment, we highlighted the limited performance improvements of InstructPix2Pix in Table 2 and Section 4.2 Main Results, attributing this to the lack of scene diversity caused by its reliance on fixed segmentation label maps.

When comparing methods that generate labels, our approach achieves better image-label alignment than DatasetDM. This improvement stems from our text-to-image generation model learning viewpoints from Cityscapes. As a result, even with the same label generator architecture, our finetuned text-to-image generation model provides representations with a smaller domain gap when generating images based on the Cityscapes dataset.

We sincerely appreciate the opportunity to perform more diverse comparisons and analyses that highlight the strengths of our approach, as well as the chance to further clarify our contributions. If you have any additional questions or concerns, please do not hesitate to let us know.

We greatly appreciate your thoughtful comments. A detailed response to your concerns is provided below, and the corresponding updates have been included in the revised manuscript. Please feel free to share any further suggestions.

[W2.1] Enhancing Writing, Technical Descriptions, and Experimental Structure

We acknowledge the importance of improving the presentation of our paper and appreciate you pointing this issue out. We reorganized the paper and made overall adjustments to parts including Sections 3 and 4, and Figures 1, 2, 3, 4 and 5. Following are the clarifications regarding the issues that the reviewer is confused with.

[LoRA Fine-Tuning] We conducted LoRA fine-tuning on Stable Diffusion XL by adding LoRA layers to all attention linear projection layers (query, key, value, and output). Since "Original LoRA" in this work refers to updating all the added LoRA layers, we have revised the description in L249 in the revised manuscript to reflect this accurately. Additionally, we labeled all figures involving the T2I model with "Attention Layers" to clearly indicate that these are the attention layers of the T2I model, including Figures 2, 3, 4, 5, and others.

[Layer Indices] The layer indices in Figures 3 and 4 represent the "attention layer indices" identified in Stable Diffusion XL. The left side corresponds to shallower layers, while the right side corresponds to deeper layers in a continuous progression. We labeled all relevant figures with "shallower" to "deeper" to provide clearer context.

[Stage 3, 4] For stages 3 and 4, we have newly included Section 3.4: Training Label Generator and Generating Diverse Segmentation Datasets to ensure our paper is self-contained, and revise minor elements of Figure 2 for clearer presentation. Additionally, we provide further implementation details for the label generator in Appendix A.1, accompanied by the newly added Figure 7, which illustrates the label decoder architecture. Below is a brief summary of the explanations for stages 3 and 4.

[Brief Summary of Stage 3] We train an additional lightweight label generator to produce a segmentation label corresponding to the image, following DatasetDM. To train the label generator, we add noise to the given labeled image and denoise the image with the fine-tuned T2I model, which can provide semantically rich intermediate multi-level feature maps and cross-attention maps. Distinct from DatasetDM, we train the label generator based on the fine-tuned T2I model using Selective LoRA. The added fine-tuning process causes a significant difference in image-label alignment, which we discussed in Appendix A.6 Comparison of Image-Label Alignment. Furthermore, due to the difference between the base T2I model, architecture details slightly changed, as described in Appendix A.1.

[Brief Summary of Stage 4] Diverse image-label pairs are generated to address both domain generalization and in-domain scenarios. For domain generalization, text prompts are modified to include adverse weather conditions (e.g., foggy, snowy, rainy, night-time) by extending the default prompt, such as "photorealistic first-person urban street view," to, for example, "in foggy weather," enhancing the model’s ability to generalize across varying environmental conditions. For in-domain scenarios, diversity is introduced by varying the class names within the prompt template (e.g., "… with car", "… with car", etc.), allowing for the generation of images that reflect different object class combinations while maintaining consistency with the in-domain characteristics.

[Organizing Experimental Section] We have reorganized the experimental section (Section 4) as the reviewer suggested. To be more specific, we explained the experimental setup and implementation details at the beginning of the experimental section. Then, we reported the main performance improvements in a section titled 'Main Results on the Semantic Segmentation Benchmarks'. After the main result, we demonstrated various analyses and ablation studies. We deeply appreciate the reviewer's valuable feedback.

As the discussion period is nearing its end, we wanted to kindly remind you about our revised manuscript.

We have carefully revised the paper and incorporated the additional experiments you requested. Furthermore, we have addressed the major concerns raised by the other reviewer.

Could you please let us know if our revisions resolve your concerns? Your feedback is crucial, and we sincerely hope our updates meet your expectations.

Thank you for your time and consideration.

Thank you for sharing your valuable feedback. Below, we provide additional responses to address the concerns raised by the reviewer.

however, as shown in the new table provided by the authors, the improvement from selective LoRA is not as significant compared to fine-tuning only some of the LoRA layers.

We have further highlighted the degree of performance improvement achieved through LoRA fine-tuning based on the performance of DatasetDM, the major baseline that utilizes the pretrained text-to-image generation model. The table allows a direct comparison of the performance gains obtained purely through LoRA fine-tuning.

Table R2.1. Comparison of Performance Improvements Across Various LoRA Fine-tuning Approaches

As shown in the table, our Selective LoRA approach achieves a performance improvement of 1.31, which is more than twice the improvement observed with hand-crafted layer selection approaches (SA-only, CA-only). Furthermore, we believe that a 1.31 mIoU improvement is significant in the context of segmentation performance. Detailed experimental results can be found in Appendix A.9: Comparison with Hand-Crafted Layer Selection Approaches in our revised paper.

Additionally, training a LoRA for a specific domain is a common practice and not particularly novel.

While LoRA fine-tuning for specific domains is widely recognized, as noted by the reviewer, no prior studies have explored its application in segmentation dataset generation approaches [1, 2, 3]. Furthermore, as demonstrated in the comparison of LoRA fine-tuning methods provided in the response above (Table R2.1), we have shown that the Original LoRA only marginally improves performance due to issues such as overfitting and memorization, which results in the generation of non-informative samples. In contrast, our Selective LoRA achieves a significant performance improvement of 1.31. This clearly highlights the substantial difference between Selective LoRA and Original LoRA.

The additional detailed technical differences with the original LoRA, recognized as a novel approach by all reviewers (B65c, hMNt, KMEN, kFPj), can be found in L254-L256 of our revised paper, as outlined below.

L254-L256: The key distinction of Selective LoRA lies in selectively fine-tuning only the crucial layers based on an automatically computed score, termed concept sensitivity, for the desired concept in the source dataset, while previous LoRA studies update all projection layers.

Thank you once again for providing valuable feedback. Although there isn’t much time remaining, if you have any additional concerns or questions, please let us know, and we will do our best to address them within the discussion period.

Reference

[1] Wu, Weijia, et al. "Datasetdm: Synthesizing data with perception annotations using diffusion models." Advances in Neural Information Processing Systems 36 (2023): 54683-54695.
[2] Wu, Weijia, et al. "Diffumask: Synthesizing images with pixel-level annotations for semantic segmentation using diffusion models." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.
[3] Nguyen, Quang, et al. "Dataset diffusion: Diffusion-based synthetic data generation for pixel-level semantic segmentation." Advances in Neural Information Processing Systems 36 (2024).

We sincerely appreciate your insightful feedback. Detailed responses to your concerns are provided below, and the revisions have been incorporated into the updated version. Please let us know if there are any further suggestions or comments.

[W1.1] Clarification of Label Generator Integration and Method Distinction

The explanation of the label generator was provided in Appendix A.1 of the original paper. However, as the reviewer pointed out, the main paper did not sufficiently explain Stage 3 (label generator). Additionally, the description of Stage 4 (generating image-label pairs) was inadequate. Therefore, we added "Section 3.4: Training the Label Generator and Generating Diverse Segmentation Datasets to explain both Stage 3 and Stage 4. Furthermore, we extended Appendix A.1 with additional technical details and added Figure 7 to illustrate the label decoder architecture. Below is a brief summary of the explanations for stages 3 and 4.

[Stage 3] We train an additional lightweight label generator to produce a segmentation label corresponding to the image, following DatasetDM. To train the label generator, we add noise to the given labeled image and denoise the image with the fine-tuned T2I model, which can provide semantically rich intermediate multi-level feature maps and cross-attention maps. Distinct from DatasetDM, we train the label generator based on the fine-tuned T2I model using Selective LoRA. The added fine-tuning process causes a significant difference in image-label alignment, which we discussed in Appendix A.6 Comparison of Image-Label Alignment. Furthermore, due to the difference between the base T2I model, architecture details slightly changed.

Specifically, given the increased number of blocks and channels in SDXL, we selected specific blocks to extract multi-scale feature maps and multi-scale cross-attention maps. Feature maps were extracted from the last feature block at each resolution of the upsampling blocks, while cross-attention maps were sampled at regular intervals (every 7 blocks) from the total 36 upsampling blocks (i.e., 1st, 8th, ... 29th, 36th). Moreover, as shown in the updated Figure 7, Stable Diffusion XL has only three resolution levels, compared to the four resolution levels in the Stable Diffusion v1.5 architecture. This difference required minor adjustments to the pixel decoder and transformer decoder, as described in Appendix A.1. Importantly, to ensure a fair comparison, the reported scores for DatasetDM were obtained using a re-implemented version based on SDXL with the same modifications.

[Stage 4] Diverse image-label pairs are generated to address both domain generalization and in-domain scenarios. For domain generalization, text prompts are modified to include adverse weather conditions (e.g., foggy, snowy, rainy, night-time) by extending the default prompt, such as "photorealistic first-person urban street view," to, for example, "in foggy weather," enhancing the model’s ability to generalize across varying environmental conditions. For in-domain scenarios, diversity is introduced by varying the class names within the prompt template (e.g., "… with car", "… with car", etc.), allowing for the generation of images that reflect different object class combinations while maintaining consistency with the in-domain characteristics.

[W1.2] Comparison in Training Efficiency with DatasetDM

As mentioned in lines L402-404 of the revised manuscript, training Selective LoRA only requires one hour on a single Tesla V100 GPU. While the reviewer is concerned about the additional stage in our method, we want to clarify that the additional stage requires a marginal amount of time compared to the entire training time (20 hours for training label generator). Considering the performance gains of using Selective LoRA compared to DatasetDM, we believe that the additional one hour of training is not as significant as the reviewer may be concerned about.

[W1.3] Semantic Segmentation Experiments on More General Datasets

In response to the reviewer's request, we conducted a new experiment for generating a semantic segmentation dataset using Pascal-VOC, a more general dataset in A.8 In-domain Experiments for the General Domain Dataset (Pascal-VOC). For this, we utilized only 100 images for segmentation, followed by a few-shot Pascal-VOC experiment in DatasetDM.

Pascal-VOC (100 images)

As shown in the table, although DatasetDM reduces performance compared to the baseline, our approach successfully enhances segmentation performance, even on a more general dataset. This experiment demonstrates the effectiveness of the selective learning approach, even with datasets exhibiting minimal distribution shifts. We include further experimental details and a qualitative comparison in Appendix A.8 and Figure 13 of the revised manuscript. Furthermore, we have added this limitation in 5. Conclusion and Future Work of our revised manuscript, and we believe that extending our method to general datasets would be interesting future work.

[W1.4, Q1.1, Q1.2] Impact of Selective Learning on Segmentation Map Quality and Quantitative/Qualitative Comparison

As the reviewer noted, the selective learning process does impact the quality of the generated segmentation maps. Following the reviewer's suggestion, we have included a new qualitative comparison in Figure 6 of the revised manuscript. This comparison demonstrated that our Style-Selective LoRA significantly outperforms both DatasetDM and the original LoRA (which applies LoRA parameters to all layers).

As illustrated in the figure, our Style-Selective LoRA significantly enhances the quality of segmentation maps compared to DatasetDM.

Additionally, while we could not discuss this in the main paper due to the page limit, we initially included such discussion in our Supplementary Appendix A.6 Comparison of Image-Label Alignment. Table 8 presents a quantitative comparison of image-label alignment between our Selective LoRA, DatasetDM, and other Selective LoRA variants. To provide quantitative results, we use the predictions from the pretrained Mask2Former model, which was fully supervised on the 100% Cityscapes dataset and achieves a 79.40 mIoU, as a proxy for the ground truth mask. Below are the key results from Table 8, which compares the image-label alignment (mIoU) of DatasetDM and ours under in-domain experiments. We selected 2% of layers for Style-Selective LoRA.

Similar to the qualitative results, the Style-Selective LoRA outperforms both DatasetDM and the Viewpoint-Selective LoRA. We attribute this to the domain gap between the pretrained text-to-image (T2I) model (SDXL) and the source dataset (Cityscapes), as discussed in Appendix A.6: Comparison of Image-Label Alignment under Analysis of the Qualitative Comparison.

[W1.5] Correction of Stage 4 of Figure 2 Annotation Error

We want to clarify that stage 4) of Figure 2 (in the original manuscript) is not reversed. Let us use the Cityscapes example as the running example. With Cityscapes, the style-sensitive layers learn the style of Cityscapes, while the viewpoint-sensitive layers learn the viewpoint of Cityscapes. This means that the style-sensitive layers can only output the style of Cityscapes, making them less effective at generating other styles of urban scenes. Conversely, the viewpoint-sensitive layers, which learn the viewpoint of Cityscapes, can generate diverse styles because the SD model retains knowledge of other styles. Therefore, to generate diverse styles, we need viewpoint-sensitive layers, and to generate diverse viewpoints, we need style-sensitive layers.

We acknowledge that the terms "style sensitive" and "viewpoint sensitive" layers might be confusing due to their seemingly opposite outputs. To address this, we have revised Figure 2 by removing the Style-Selective LoRA and focus on the segmentation dataset generation framework. Furthermore, we added qualitative results for the Viewpoint-Selective LoRA with a brief explanation in the motivation figure (Figure 1) to better illustrate the sensitivity.