CoSimGen: Controllable diffusion model for simultaneous image and segmentation mask generation

We thank the reviewer for the thoughtful comments. We clarify the key points as follows:

1. On Novelty vs. MosaicFusion and Related Works We agree MosaicFusion [1] and similar works exist, but their objectives and mechanisms differ fundamentally from ours. For instance: a.) MosaicFusion composites object masks for data augmentation, whereas CoSimGen performs joint diffusion-based generation of both the full image and its segmentation mask under explicit controllable conditions. b.) Prior mask-to-image methods [2,3] rely on mask inputs; CoSimGen uniquely supports flexible conditioning (class vectors or text prompts) without requiring any mask input, which is critical for data-scarce domains. c.) Our Spectron mechanism introduces spatial vs. spectral conditioning, which is a principled design separating semantic and timestep conditioning, which prior works do not address. These are novel technical contributions of our work.

2. On SOTAs and Experimental Scope Our primary goal is to solve the underexplored problem of paired controllable generation, not general image synthesis. Therefore, we selected baselines commonly adopted for paired generation. State-of-the-art image synthesis models like SD [6] and DiT [4] excel at image generation only, but lack mechanisms to ensure segmentation-mask consistency under conditioning. Our contributions target this gap rather than raw image fidelity alone. This positioning explains why comparisons to generic large-scale image generators were not emphasized.

3. On Super-Resolution vs. Latent Diffusion We chose a two-stage pipeline to stabilize structural alignment of masks before refining textures, a well-justified design choice for generating paired outputs where structural accuracy is critical. This is supported by analysis already in the paper (Sec. 3.2.3, Sec. 5.4, Fig. 6 and appendix), and aligns with the goal of preserving segmentation quality at high resolution—something direct high-resolution generation methods often compromise.

4. On Technical Details and Related Work We acknowledge the suggestion to expand related work with DiT [4] and SD [6]. Our conditioning strategy differs because it is tailored for multi-perspective paired generation, which these works do not address. Section 3.2 will be clarified to make these distinctions explicit. The SR module details (loss and structure) are also described in the appendix for reproducibility.

We are sure that CoSimGen addresses a distinct and practically important problem, which is a controllable simultaneous image and mask generation, with novel conditioning mechanisms and a design optimized for paired consistency, which current SOTA image synthesis methods do not provide. This will provide a new and useful approach to image generation controllability.

We appreciate the reviewer's feedback and address each concern as follows:

1. On Novelty and Prior Work We acknowledge existing works like DatasetDM, MaskFactory, and Tumor Synthesis, which are valuable contributions. However, these methods either (i) focus on generating data for specific segmentation contexts (e.g., dichotomous or tumor segmentation) or (ii) lack controllability across text, class, and temporal embeddings. CoSimGen’s contribution is a unified controllable diffusion framework where semantic control (text, class) and structural control (spatial-spectral conditioning) operate jointly. This allows dynamic switching between text and class prompts and maintains paired consistency throughout the denoising process. This alone is a capability not demonstrated in cited works.

2. On Research Gap, Baselines and Comparisons We appreciate the emphasis on baseline evaluation. Our primary goal is to address the underexplored problem of simultaneous controllable image–mask generation, which differs from generic image synthesis. Existing diffusion-based models (e.g., ControlNet, Paint-by-Example) and GAN-based methods largely target either image generation only or mask-to-image translation, without mechanisms for bidirectional controllability (e.g., switching between text and class prompts) or for ensuring paired structural alignment throughout the diffusion process. Given the absence of prior works designed specifically for controllable joint generation, we selected commonly adopted paired-generation baselines (GAN and VAE families) to establish a fair and meaningful comparison for this problem setting. Our experimental results consistently show CoSimGen improves both mask consistency and semantic fidelity over these approaches, validating the impact of our conditioning mechanisms rather than mere architectural scale. We also emphasize that ablation results already included in the paper isolate the contributions of Spectron, Textron, and the SR module (Table 4). These results demonstrate that each module contributes significantly (e.g., noticeable Dice improvements when adding Spectron and Textron), reinforcing that performance gains are due to the proposed design, not pipeline complexity. While modern diffusion models like DiT or FLUX excel at high-quality image synthesis, they do not address paired controllability with consistent segmentation alignment, which is the unique contribution of our work.

3. On Two-Stage vs Single-Stage Generation We deliberately use a two-stage pipeline for stability in joint image–mask generation. Direct high-resolution paired generation via latent diffusion degraded segmentation accuracy and boundary alignments. Our pipeline allows structural alignment first, then texture refinement, which is crucial for clinical/structured domains. This decision is empirically justified and supported by ablations (added to the supplement).

4. Underlying Model Clarity and Reproducibility We clarify: CoSimGen builds on pixel-space DDPM backbone for the base stage and uses a super-resolution diffusion model for refinement. This ensures precise spatial alignment for segmentation masks while achieving high-resolution outputs. Full architecture and hyperparameter details are included in the main manuscript and appendix for reproducibility. Code will also be made public.

5. On Figures and Presentation We acknowledge the presentation concerns. Most of the figures are vector graphics. The few which are not are due to the heaviness of the figures when rendering the pdfLaTex. We would update all the figures with vector graphics, consistent fonts, and improved diagrams for clarity in the final version of the manuscript.

Thank you for acknowledging our contributions, particularly the novelty of Spectron and Textron. We address your concerns below:

1. On Figure 1 Mask Visualization The dual coloring in Fig. 1 reflects boundary-enhancement visualization used for clarity, not an error in mask generation. The actual mask is a single-class binary representation internally. We will clarify this in the figure caption.

2. Related Work Coverage We appreciate the suggested references. While our work focuses on simultaneous controllable image–mask generation, we will incorporate additional discussion of related frameworks such as SemFlow, Emu Edit, InstructDiffusion, and UniGS to position CoSimGen within the broader context of diffusion-based segmentation and editing. These additions strengthen, rather than diminish, the novelty of our perspective-conditioned approach.

3. Why Not Latent Diffusion or One-Stage Generation? Our decision for a two-stage design (base + super-resolution) was intentional: a.) Stability in Joint Generation: Directly generating paired 512×512 outputs in a single latent stage significantly increases optimization complexity for dual outputs. Empirically, this approach increases computational overload and harder to optimize especially with higher diffusion step requirement for the joint image-mask denoising. b.) Two-Stage Pipeline Advantage: It allows structural alignment to stabilize first at low resolution before refining texture details, yielding superior mask fidelity. We will add these ablation results to the supplementary.

4. Training and Inference Efficiency CoSimGen’s two-stage design achieves competitive efficiency: As mentioned in the manuscript, it trains for approximately 72 hours on 1×H100 for 500k steps. Detailed training time and inference speed comparisons with other methods will be added to the revised paper.

5. Capability vs. Latest Latent Diffusion or DiT Current latent diffusion or DiT-based models primarily target image generation only, without explicitly ensuring mask consistency under controllable conditions. In contrast, CoSimGen introduces perspective-conditioned conditioning (Spectron + Textron) to guarantee joint consistency. This is, indeed, a capability the previous models lack. Incorporating such conditioning into latent models is a promising future direction, but remains unexplored. We believe the introduced mechanisms and design trade-offs represent meaningful contributions beyond current alternatives.

We appreciate your detailed feedback. We respectfully address the concerns as follows:

1. On Novelty and Contribution While diffusion-based frameworks and contrastive learning individually exist, our contribution is not a mere combination but a new conditioning formalism tailored for simultaneous image–mask generation: a task underexplored in current literature.

In our work, Spectron introduces dual-domain conditioning (spatial vs. spectral) that exploits the orthogonality of space and diffusion timesteps to disentangle semantic and temporal signals. This design is neither obvious nor standard in diffusion models, which typically inject all conditions through a single channel.

Additionally, Textron unifies text-class alignment within the generation loop rather than as a pre/post-hoc step, enabling controllability without fine-tuning the backbone.

Together, these mechanisms enable paired controllable synthesis without altering the diffusion backbone. This is a lightweight yet effective architectural innovation with direct practical benefits for low-resource domains.

2. On Baselines and Experimental Rigor We acknowledge the importance of fair baselines. Our primary goal was to benchmark against widely adopted paired-generation methods (GAN and VAE families), as no prior diffusion method explicitly addresses simultaneous image–mask generation with controllability. Existing diffusion-based models (e.g., ControlNet, Paint-by-Example) and GAN-based methods largely target either image generation only or mask-to-image translation, without mechanisms for bidirectional controllability (e.g., switching between text and class prompts) or for ensuring paired structural alignment throughout the diffusion process.

Given the absence of prior works designed specifically for controllable joint generation with hot-swapping of prompt modality, we selected commonly adopted paired-generation baselines (GAN and VAE families) to establish a fair and meaningful comparison for this problem setting. Our experimental results consistently show CoSimGen improves both mask consistency and semantic fidelity over these approaches, validating the impact of our conditioning mechanisms rather than mere architectural scale.

Moreso, ablation studies (Table 4) isolate Spectron, Textron, and the super-resolution module, showing complementary contributions (Spectron +8% Dice, Textron +6% Dice over baseline).

3. On Writing and Figure Consistency We acknowledge minor inconsistencies in notation and Figure 3. These have been corrected for clarity. The figure now explicitly matches the textual description of embeddings.

Our work, CoSimGen, provides a principled framework for controllable paired generation, introducing perspective-conditioned diffusion that substantially improves both semantic fidelity and structural alignment in scenarios where annotated data is scarce. We believe these contributions are significant and address a critical gap in generative modeling for vision tasks and would contribute immensely to the development of future versatile free-style-prompting generative AI.