MaskFactory: Towards High-quality Synthetic Data Generation for Dichotomous Image Segmentation

We sincerely thank the reviewer for their meticulous review and helpful suggestions. We address each point as follows:

Q1: Typos and Errors

We apologize for these oversights and will correct all identified issues in the revised version:

a. We will change (V,E) to (V, E_s) in line 140 and ensure consistency throughout Section 3.2.2. b. We will unify the font style across the paper, including lines 139-140 and Algorithm 1. c. We will revise Figure 2 and Section 3.3 to accurately reflect our method's implementation, clarifying the use of text prompts, segmentation masks, and edge conditions. We will also provide a detailed description of {p^m_i} for consistency. d. We will correct "Topylogy" to "Topology" in Figure 2. e. We will remove full stops in the titles of Section 4, 4.1, and 4.2.

Q2: Canny Condition Usage

We appreciate this insightful observation. While the canny condition is indeed implicitly included in the mask, our experiments show that explicitly including it improves results. We have added an ablation study to demonstrate this:

These results show that including the Canny condition leads to better performance across all metrics. In Figure 3 of our global PDF, we demonstrate the effect of adding the Canny condition. We found that incorporating Canny edges provides better constraints on the boundaries of generated images, resulting in more precise and detailed outputs.

Q3: Comparison with SAM-HQ

While our improvement over SAM-HQ is modest, it's important to note that SAM-HQ has already incorporated a large amount of additional data during its fine-tuning process. This likely means that SAM-HQ's performance is approaching a plateau, making significant improvements more challenging. In contrast, our method shows more substantial improvements over other baselines, demonstrating its effectiveness, especially in scenarios where performance hasn't yet reached saturation.

Q4: Suggestions

1. ImageDream: We have explored ImageDream and downloaded the weights as suggested. However, we found that it may not be directly suitable for binary image editing without fine-tuning. The results are shown in Figure 1 of our global PDF. Despite this, the method is indeed very inspiring for our work, and we will include a reference to it in our related work section.

2. Topology visualization: We agree this would enhance the paper. We have included topology structure visualizations for selected samples in Figure 2 of our global PDF. These visualizations clearly demonstrate how our method preserves and manipulates topological structures during the editing process. We will incorporate these more intuitive visualizations in the next version of our paper.

3. Table 1 formatting: We will adjust the spacing in Table 1 for better readability.

4. Related work discussion: We appreciate these suggestions and will expand our discussion of [2] and [3]. We will explore incorporating the data filtering strategy from [2] and the dataset augmentation techniques from [3] into our work, and report the results in the revised version.

We acknowledge that some aspects of the paper need refinement, and we are committed to addressing all these issues in our revision. We will ensure that our method description accurately reflects its implementation, including the use of text prompts, masks, and canny edges in the generation process. We will revise Figure 2, Section 3.3, Formula 8, and the appendix algorithm accordingly.

We appreciate the opportunity to clarify these points and look forward to submitting a significantly improved version of the paper.

References:

[1] Wang, P., & Shi, Y. (2023). Imagedream: Image-prompt multi-view diffusion for 3d generation. arXiv preprint arXiv:2312.02201.

[2] Yang, L., et al. (2024). Freemask: Synthetic images with dense annotations make stronger segmentation models. NeurIPS 36.

[3] Zhu, L., et al. (2024). Generative Enhancement for 3D Medical Images. arXiv preprint arXiv:2403.12852.

Thank you very much for your valuable feedback, time, and attention. We sincerely appreciate your encouragement. We will continue to refine the descriptions in our document to make them even clearer and more accurate. If you have any further questions or need additional clarification on any point, please don't hesitate to let us know.

We greatly appreciate the reviewer's comprehensive examination of our work. Our responses to each point are as follows:

Q1: Novelty of the method and combination of existing approaches

While our approach does integrate existing methods, its novelty lies in:

1. Task-specific adaptations for DIS tasks: Our method is specifically tailored for Dichotomous Image Segmentation, addressing unique challenges such as high-precision requirements and the need for diverse training samples.

2. Synergistic integration of rigid and non-rigid editing: We combine these techniques in a novel way to provide comprehensive mask editing capabilities, enhancing both geometric precision and topological consistency.

3. Topology-preserving adversarial training: This innovative approach ensures structural integrity during non-rigid editing, which is crucial for maintaining the quality of binary masks in DIS tasks.

These innovations collectively contribute to a method that is greater than the sum of its parts, specifically designed to meet the demands of DIS tasks.

Q2: Validation of generated data boosting DIS performance

We have rigorously validated the effectiveness of our generated data in improving DIS performance:

1. Our method achieves a 5.6% improvement in maxF1 score over the baseline, demonstrating significant performance gains.

2. We employ multi-conditional control generation to enhance consistency between generated images and masks, addressing the concern of unrealistic images.

3. The combination of rigid and non-rigid editing increases data diversity, mitigating the issue of limited variability.

To further illustrate the impact of our generated data, we present additional experimental results:

These results clearly demonstrate that our generated data significantly improves performance across multiple metrics.

Q3: Limited evaluation on one dataset

We acknowledge this limitation in our initial submission. To address this, we have conducted additional experiments on various datasets to demonstrate the generalizability of our approach. These results will be included in the revised manuscript.

Q4: Applicability to general semantic segmentation datasets

We have explored the applicability of our method to other semantic segmentation tasks using InSPyReNet [1] as a baseline. Here are some results:

1. ECSSD (Edge detection):

   Method	S_α ↑	F_max ↑	MAE ↓
   Baseline	0.942	0.959	0.023
   Ours (Mix1000)	0.950	0.962	0.022

2. HKU-IS (Salient object detection):

   Method	S_α ↑	F_max ↑	MAE ↓
   Baseline	0.931	0.952	0.022
   Ours (Mix1000)	0.943	0.954	0.021

3. UOD (Underwater object detection):

   Method	S_α ↑	F_max ↑	MAE ↓
   Baseline	0.868	0.878	0.067
   Ours (Gen1000)	0.883	0.901	0.056

These results demonstrate the effectiveness of our approach across various semantic segmentation tasks. We are currently extending our experiments to more challenging scenarios such as NI-seg and FI, and will include these results in the revised manuscript.

Reference

[1] Yu, Q., et al. "Multi-view Aggregation Network for Dichotomous Image Segmentation." CVPR, 2024.

We are grateful for the reviewer's perceptive observation. We provide a more in-depth analysis below:

Q: The paper acknowledges issues with unnatural images in complex scenarios, but could delve deeper into how these might impact practical applications.

A: We address this concern by focusing on two key points:

1. DIS data scarcity for VAE training: We opted for a more general VAE to ensure better generalization. Additional experiments show:

   VAE Training Data	maxF1 ↑	M ↓	E_φ^M ↑
   General dataset	0.784	0.073	0.875
   Small DIS dataset	0.762	0.079	0.853

   A DIS-specific VAE produces more visually coherent images but leads to performance decrease due to overfitting.

2. Impact on performance: Despite occasional unnatural images, our method significantly improves downstream segmentation performance. We achieve a 5.6% improvement in maxF1 score compared to the baseline, suggesting increased data diversity outweighs drawbacks of some unnatural samples.

We sincerely appreciate the reviewers' thorough evaluation and constructive feedback. We are grateful for the opportunity to clarify and expand on our work.

Q1: Can you elaborate on the methodology, especially regarding mask editing and unique designs?

We will expand the methodology section in the revised manuscript to include:

1. Rigid Editing: We will provide a comprehensive explanation of our rigid editing process using Zero123. As stated in Section 3.2.1: "We leverage the Zero123 method, which employs a viewpoint-conditioned diffusion model ψθ to manipulate masks' perspectives." We will elaborate on how this allows for precise geometric adjustments, focusing on viewpoint and scale transformations.

2. Non-rigid Editing: We will offer a more detailed description of our non-rigid editing process, including:

   • Topology-preserving adversarial training
   • Mutual attention mechanism
   • Foreground-background focus guidance

   As mentioned in Section 3.2.2: "We introduce a topology-preserving adversarial training mechanism to mitigate artifacts and structural degradation in binary mask editing." We will expand on how this mechanism works and its importance in maintaining structural integrity.

3. Multi-conditional Control Generation: We will provide a more in-depth explanation of our image generation stage, as described in Section 3.3: "We introduce a multi-condition control generation method to achieve precise RGB image generation."

We will also include architectural diagrams and pseudocode to enhance clarity and understanding of our methodology.

Q2: Can you clarify your contributions beyond existing methods?

Our work presents several key innovations in the field of mask editing and dataset generation for DIS tasks:

1. Specific Improvements for DIS Tasks: Our method is designed for Dichotomous Image Segmentation (DIS) tasks, which require extremely high fidelity and precision in mask editing. Our approach generates diverse samples while maintaining high accuracy.

2. Methodological Innovations:

   • Topology-preserving adversarial training for non-rigid editing, ensuring structural integrity of edited masks
   • Combination of rigid and non-rigid editing techniques, providing comprehensive mask editing capabilities
   • Multi-conditional control generation method for precise image-mask pair creation

3. Performance Improvements: As demonstrated in Table 2 of our paper, our method consistently enhances the performance of various state-of-the-art segmentation networks across multiple datasets, proving its effectiveness and generality.

4. Addressing Unique Challenges in DIS Tasks: Our method addresses the need for high-precision annotations and diverse training samples in DIS tasks, which has rarely been specifically targeted in previous methods.

Q3: Can you provide more comprehensive ablation study results?

We have conducted additional experiments to provide a complete picture of the impact of each loss component:

These results demonstrate that while L_GAN alone provides a strong baseline, the combination of all three loss components yields the best performance. This underscores the importance of our multi-faceted approach to mask editing and image generation.

Q4: Can rigid and non-rigid mask editing be integrated into one framework?

Currently, we have not integrated these two frameworks, primarily because the rigid editing part cannot use topology preservation constraints. After viewpoint transformation, the detected key points undergo significant shifts, making topology preservation challenging.

In future work, we plan to explore the following directions to attempt integration:

• Develop adaptive loss functions that can handle both rigid and non-rigid transformations
• Design multi-stage editing processes that handle rigid and non-rigid edits separately while maintaining overall consistency
• Research novel architectures that can automatically select appropriate editing techniques based on the input and desired transformation

Q5: How do you plan to address the limitations mentioned in the paper?

We are actively addressing these issues, particularly focusing on the problem of unnatural images:

1. Unnatural Images: We recognize that this issue primarily stems from VAE training. We attempted to train a specialized VAE using the DIS dataset, but due to limited data, the results were not as effective as using a larger general dataset. Here are our preliminary results:

   VAE Training Data	maxF1 ↑	M ↓	E_φ^M ↑
   General dataset	0.784	0.073	0.875
   Small DIS dataset	0.762	0.079	0.853

   In the future, we plan to explore:

   • Data augmentation techniques to effectively increase the size of the DIS dataset
   • Transfer learning methods, using VAEs pre-trained on large-scale datasets and then fine-tuned on DIS data

2. Inaccuracies in Complex Scenarios: We are developing hierarchical generation approaches and improved attention mechanisms to better capture long-range dependencies in complex scenes.

3. Reliance on Pre-annotated Pairs: We are researching self-supervised learning methods and weak supervision strategies to reduce the need for extensive pre-annotated data.

We will include preliminary results and discussions of these ongoing efforts in the revised manuscript.