We sincerely thank the reviewer for the insightful and constructive feedback. Our detailed responses are provided below.

Response

Response to weaknesses

W1: In Tab. 1, Seg2Any outperforms previous methods significantly, but in Tab. 2, its performance on generation metrics (FID) is poor, and it also does not perform well on COCO MIoU. This raises concerns about the effectiveness of the method as well as the validity of the benchmark itself. Authors are encouraged to compare under fair settings. For example, how will the results of Tab.1 and Tab.2 change when other baseline methods, especially FreestyleNet and PLACE, are initialized by flux?

Thanks for raising this insightful question.

On the relevance of FID. Recent work[2] has shown that FID correlates weakly with human perceptual preference. FID provides a reference for quality evaluation to a certain extent, but it is not necessarily the best evaluation indicator.

Fair comparison under FLUX. As the official PLACE training recipe has not been released, we cannot reproduce its auxiliary losses and its additional training data. To ensure a fair comparison, we reimplement only the essential PLACE attention mask under the FLUX.1 dev architecture while keeping all other training details identical to ours: learning rate 1e-4, batch size 16, 20k steps, 4 NVIDIA H100 GPUs, etc. Regarding FreestyleNet under FLUX initialization, our Semantic Alignment Attention Mask (SAA) is the direct implementation of FreestyleNet’s attention mask within the MM-DiT framework. The corresponding metrics are already reported in Table 3 row 1 and Table 4 row 1.

Table A: Supplementary experiments of FreestyleNet and PLACE with flux initialization on COCO Stuff and ADE20K. Bold represent the best.

Table B: Supplementary experiments of FreestyleNet and PLACE with flux initialization on the SACap-Eval benchmark. Bold represents the best.

As the above Tables A and B show, under identical FLUX initialization and training settings, our model outperforms PLACE attention by 2.9% MIoU on ADE20K and by 4.1% MIoU on SACap-Eval, confirming the superiority of our design in a strictly fair comparison.

W2: The results of the ablation experiment are confusing. Comparing the results of the first row in Tab. 4 with those in other rows, it appears that some designs negatively impact the final performance.

Thank you for highlighting the sub-optimal metrics in the final configuration. In the final version, we will incorporate the following explicit justifications for each component:

• SSFA is essential because it delivers a remarkable +4.7% boost in class-agnostic mIoU in Table 4, directly addressing our primary objective of precise shape preservation.
• AIA improves the average of four region-wise metrics by 1.4% in Table 4 and effectively prevents attribute leakage. We will add new visual results in the final version to make this benefit concrete.
• Training-based AIA is essential for recovering global coherence, as evidenced by realistic reflections and shadows.
• CITF is retained solely for efficiency. Standard metrics (FID, CLIP, IR, and Pick) imperfectly capture perceptual quality. We will provide extra visual results in the final version confirming CITF leaves image fidelity almost unchanged.

Response to questions

Q1: I am curious about how the model would perform for entities of different sizes. As the entity contour map is down-sampled, how can we ensure the distinction between dense entities?

Thank you for this valuable question. As illustrated in Figure 1 row 2, Seg2Any preserves the fine vertical bars on the railing, demonstrating robustness to small-scale objects. Additionally, Appendix Figure 6 (last row column 4) shows that our method performs equally well in dense scenes. Due to the NeurIPS 2025 rebuttal policy, we cannot include new visuals here. In the final version, we will include additional examples such as around 20 ceiling lights smaller than 64×64 pixels and densely overlapping scenes (e.g., 19 fruits across nine categories, 12 cars of distinct colors, and five heavily occluded football players), where our method continues to yield accurate boundaries. We acknowledge that extreme density can still induce object merging. Therefore, we will also present failure cases to transparently report these limitations.

Q2: How does performance change if the model is initialized using Stable Diffusion[1]?

Thank you for this question. However, Seg2Any is specially designed for MM-DiT:

• Seg2Any assigns distinct attention masks to different layers: Attribute Isolation Attention Mask (AIA) on FLUX layers 20–38 for instance rendering, Semantic Alignment Attention Mask (SAA) elsewhere for global coherence. Such layer-wise scheduling is absent in Stable Diffusion (SD).
• Condition Image Token Filtering (CITF) discards negligible condition tokens via token-level pruning within DiT’s unified sequence. However, SD’s separate encoder blocks preclude this filtering.

Owing to the tight rebuttal schedule, adapting Seg2Any to Stable Diffusion would demand a complete redesign of the attention mask schedules and Entity Contour Map injection pipeline, we will complete this ablation during the discussion period. The planned setup employs SD 1.5, injects the entity contour map via a ControlNet branch, and applies attention mask strategy to the U-Net cross-attention layers. We will benchmark this setup on COCO-Stuff and ADE20K. Please let us know whether this setup meets your intent during the discussion period, and we will run the experiments immediately upon confirmation.

[1] High-Resolution Image Synthesis with Latent Diffusion Models. CVPR, 2022.

[2] Rethinking FID: Towards a Better Evaluation Metric for Image Generation. CVPR, 2024

We sincerely thank the reviewer for the insightful and constructive feedback. Our detailed responses are provided below.

Response

Response to weaknesses

W1: Both the Semantic Alignment Attention Mask and the Attribute Isolation Attention Mask resemble existing techniques for layout-controllable generation [1, 2]. How does your Attribute Isolation Attention Mask differ from MIGC’s [3] attention-mask design?

The key distinction lies in the architectural target: whereas MIGC [3] and prior works [1,2] devise masks for U-Net’s cross-attention and self-attention, our Semantic Alignment Attention Mask (SAA) and Attribute Isolation Attention Mask (AIA) are designed for the multi-modal attention in the DiT framework. Moreover, MIGC applies two attention masks (instance-intra and instance-inter) within each layer. In contrast, Seg2Any assigns distinct attention masks to different layers. Our attention mask strategy aligns with FLUX’s internal hierarchy: layers 20–38 specialize in instance-level rendering, so AIA is applied only there, while layers 1–19 and 39–57, which encode global scene context, employ the SAA to preserve overall coherence.

W2: In the introduction section, the authors claim that the downsampling ratio amplified the loss of spatial information. My question is, how do authors mitigate this problem? In my opinion, the attention mask and condition tokens encoded by VAE are still sparse and lack spatial detail.

In fact, we address the loss of spatial information by introducing the Entity Contour Map as condition tokens. The attention mask is produced by a 16× nearest-neighbour downsample of the input segmentation mask—an irreversible, lossy compression that discards spatial detail. Our remedy is to introduce the Entity Contour Map as an additional condition token. Leveraging the VAE’s high-fidelity, near-lossless compression, the Entity Contour Map preserves fine spatial details. As evidenced in Table 4 (row 1 vs. 2) and Table 3 (row 1 vs. 2), the injection of the Entity Contour Map yields a clear MIoU improvement.

W3: The paper lacks qualitative ablation results for CITF. Because automatic metrics may not fully reflect human perception, I recommend adding visual comparisons to illustrate CITF’s impact.

Thank you for highlighting this point. Owing to the NeurIPS 2025 rebuttal policy, we are unable to provide new visuals during the rebuttal phase. We will include qualitative ablations comparing results with and without Condition Image Token Filtering (CITF) in the final version to illustrate its perceptual impact.

W4: Could authors further clarify the practical value of the mask-to-image generation task? In real-world applications, semantic masks can be difficult for users to obtain.

Thank you for raising this important practical concern. First, precise spatial control makes Seg2Any well-suited for design workflows such as interior or furniture layout, where users can sketch and instantly visualize the result. Second, our pipeline can automatically create large, labelled datasets for downstream segmentation tasks, drastically reducing manual annotation costs when adapting detectors to new domains or rare object categories.

[1] MultiBooth: Towards Generating All Your Concepts in an Image from Text

[2] MS-Diffusion: Multi-subject Zero-shot Image Personalization with Layout Guidance

[3] MIGC: Multi-Instance Generation Controller for Text-to-Image Synthesis

We sincerely thank the reviewer for the insightful and constructive feedback. Our detailed responses are provided below.

Response

Response to weaknesses

W1: The framework is quite complex, with multiple custom components (two attention masks, a new condition type, etc.) that need to work in concert.

Thank you for the feedback. We respectfully clarify that Seg2Any is concise: it introduces only four lightweight components: Semantic Alignment Attention (SAA), Attribute Isolation Attention (AIA), Sparse Shape Feature Adaptation (SSFA), and Condition Image Token Filtering (CITF). In addition, it requires only two inputs: free-form text and segmentation masks. Each component serves a distinct, non-redundant purpose:

• SAA guarantees that every regional caption binds to its mask without extra trainable encoders.
• AIA prevents attribute leakage across entities by masking cross-entity self-attention in mid-layers. This is critical when multiple entities share similar textures or colors.
• SSFA injects high-frequency boundary information via an Entity Contour Map, yielding a pronounced +4.7% absolute gain in class-agnostic MIoU in Table 4.
• CITF removes zero-value tokens from the Entity Contour Map, thereby shortening the token sequence and accelerating both training and inference, as illustrated in Appendix Figure 1.

W2: The method's performance on one of the closed-set benchmarks (ADE20K) is strong but still beaten by a prior method (PLACE) on the MIoU metric.

Thank you for the comment. The official PLACE training recipe has not been released. Its paper reports using the ADE20K dataset augmented with additional text–image pairs, which may explain the gap. To ensure a fair comparison, we conducted an ablation experiment: we reimplemente only PLACE’s core attention mask under FLUX.1-dev, keeping other hyperparameters fixed: learning rate 1e-4, batch size 16, 20k steps, 4 NVIDIA H100 GPUs, etc.

Table A: Supplementary experiments reimplement PLACE within the FLUX architecture on COCO-Stuff and ADE20K. Bold represents the best.

Table B: Supplementary experiments reimplement PLACE within the FLUX architecture on the SACap-Eval benchmark. Bold represents the best.

As shown in above Tables A and B, our model surpasses this PLACE attention variant by +2.9% MIoU on ADE20K and +4.1% MIoU on SACap-Eval, confirming the superiority of our design in a strictly fair comparison.

W3: The new dataset relies on a VLM for automatic captioning, which, as the authors acknowledge, can introduce label noise.

We provide indirect evidence that the impact is limited.

1. Qualitative inspection: Appendix Figure 5 shows some hard samples of SACap-1M annotations, where Qwen2-VL-72B produces accurate, fine-grained descriptions.
2. Downstream robustness: Figure 3 column 1, Figure 4, and Appendix Figure 6 demonstrate that Seg2Any still achieves precise attribute and spatial control, indicating that the model tolerates residual noise.

Response to questions

Q1: The Attribute Isolation Attention Mask is applied only to the middle layers (20-38) of the FLUX model. What is the reasoning behind this specific layer range? How does performance change if you apply it to fewer or more layers?

Thanks for this insightful question. The choice of layers 20–38 is motivated by DreamRenderer, which identifies these middle layers as primarily responsible for instance-level rendering, whereas shallower (1–19) and deeper layers (39–57) mainly encode global scene context. To verify this, we conducted a systematic case study with varying layer ranges:

• Extending AIA into layers 1–19 or 39–57 disrupts global image quality, yielding noticeable artifacts.
• Restricting AIA to only layers 20–29 or 30–38 markedly weakens attribute isolation.

We will include these ablations in the final version.

Q2: You mention that the shape guidance strength γ can be modulated for flexible, scribble-style control. The examples in the paper showcase excellent results with precise masks. Could you provide a visual example of the model's performance with a looser, scribble-style input?

In fact, Appendix Figure 2 (bottom row) already presents qualitative results obtained with $\gamma = 0.2$, where the input consists of coarse, scribble-style strokes rather than precise masks. Under the scribble-style input, Seg2Any still generates entities that respect the rough spatial layout while allowing natural shape variations, confirming the flexibility of our modulation strategy.

Q3: The paper acknowledges that using a VLM to create the SACap-1M dataset inevitably introduces some annotation noise. Have you done any analysis on the nature or rate of this noise, and how might it be affecting the model's training and final performance?

We appreciate the reviewer's concern regarding annotation noise. Quantitatively estimating the noise rate across 5.9 M region captions is non-trivial. Instead, we provide indirect evidence that the impact is limited.

1. Qualitative inspection: Appendix Figure 5 shows some hard samples of SACap-1M annotations, where Qwen2-VL-72B produces accurate, fine-grained descriptions.
2. Downstream robustness: Figure 3 column 1, Figure 4, and Appendix Figure 6 demonstrate that Seg2Any still achieves precise attribute and spatial control, indicating that the model tolerates residual noise.

Q4: The results shown are very impressive. Could you discuss or show some common failure cases for Seg2Any? For example, does it struggle with an extremely high number of objects, very complex overlapping shapes, or abstract text prompts that are hard to visualize?

Thanks for prompting a deeper discussion of failure cases. Due to the NeurIPS 2025 rebuttal policy, we cannot include new visuals here. All examples below will appear in the final version.

1. Extreme object density. Extensive tests with 19 overlapping fruits across nine categories, 12 cars of distinct colors, and 5 heavily occluded football players demonstrate Seg2Any’s strong robustness under high overlap. However, pushing the limit further, we evaluate a crowded cinema scene containing roughly 50 people and observe that the model begins to merge adjacent masks, thereby revealing a practical upper bound on the number of instances.
2. Highly abstract prompts. For abstract prompts like "a fantastical world split into three distinct zones: desert, ice, and volcano." Seg2Any yields unnatural transitions at mask seams, likely because SACap-1M contains few surreal, zone-abrupt scenes.

We sincerely thank the reviewer for the insightful and constructive feedback. Our detailed responses are provided below.

Response

Response to questions

Q1: According to the Ablation study in Table 3 and Table 4, the Condition Image Token Filtering (CITF) seems to only sightly improve the class-agnostic MIoU while negatively affecting other metrics. Statistically, the overall contribution of CITF is negative, but authors claimed "Table 4 ... Condition Image Token Filtering (CITF) leaves performance unchanged" and "CITF does not affect performance (see Table 3)". More explainations are needed here.

Thank you for pointing out the lack of rigor in our previous statements regarding the Condition Image Token Filtering (CITF). We agree that it is imprecise to claim that CITF "does not affect performance" or "leaves performance unchanged." Upon closer inspection of Tables 3 and 4, there are indeed minor quantitative influences in certain metrics. We acknowledge that overall metrics like FID, CLIP, IR, and Pick may not fully reflect human perceptual quality. We will provide more visual examples in the final version to show that CITF does not noticeably impact image quality. Our main motivation for using CITF is to accelerate training and inference while maintaining similar performance, as shown in Figure 1 of the supplementary material. We will revise to clarify these points and highlight both the efficiency gains and the minimal effect on generative quality.

Q2: Figure 3 column 4 does not support the statement in caption: "direct application of the mask without training leads to visual inconsistencies, manifesting as unnatural shadows and reflections". Shadows of the plane in Training-free AIA and Training based AIA results are both unatural, even worse than W/O AIA.

Thank you for the careful inspection. We agree that the shadow under the plane in Figure 3 column 4 is too subtle to clearly demonstrate the benefit of training the Attribute Isolation Attention Mask (AIA). In the final version we will replace this instance with a more salient example.

Q3: What is the definition of 'attribute' in the term 'attribute leakage'? Authors claims that failures in Figure 3 Row 1 are caused by 'attribute leakage', but it seems that the biggest disadvantage of W/O AIA w.r.t Training-based AIA is the text generation. One of the most impressive advanatages of Seg2Any in Figure 4 is also good text generation. However, all these experiments can not prove the contribution of AIA, because it is entirely possible that this improvement comes from other modules. Stronger evidence is needed.

Thank you for highlighting the insufficiency of our evidence. We define "attribute" as instance-level visual properties including color, texture, style and shape. In Figure 3 column 1, the letter on each badge is also an attribute, as explicitly requested by the prompt "circular badges labeled A–T". Owing to NeurIPS 2025 policy we cannot include new figures in the rebuttal, but we will add four new study cases in the final version to prove the contribution of AIA:

1. Three animals—Lego-style lion, ice-style penguin, flame-style eagle—where without AIA the Lego texture leaks to the penguin and eagle.
2. Nineteen fruits. Without AIA, the red/green apples swap colors and the kiwi incorrectly takes on dragon-fruit hues.
3. Four cats—Maine Coon, orange tabby, calico, Sphynx—where fur color and ear/eye shapes are mixed without AIA.
4. Three people wear wool and leather. Without AIA, the wool texture appears on the leather jacket.

Q4: The AIA prevents one entity token attending to other entity tokens, background tokens and global text token. Wouldn't this strategy undermine the global perception and long-term dependences among entities?

Your concern is well-taken but is already mitigated in our design. The Attribute Isolation Attention Mask (AIA) is applied only to the middle layers (layers 20–38) of the 57-layer FLUX model, which are responsible for instance-level rendering. All other layers retain full self-attention among entity, background, and global-text tokens, thereby preserving global context and long-range dependencies. Indeed, the realistic shadows and reflections shown in Figure 3 (columns 2–4) empirically confirm that our training-based AIA does not compromise long-term dependences among entities.

Q5: 5. Authors should provide attentive map or visualize some results in ablation study section. In particular, it is better to show how the results of attentive map changes with respect to different configurations of key components.

Thank you for this helpful suggestion. Owing to NeurIPS 2025 policy we cannot include new figures in the rebuttal, but the camera-ready version will add visualizations of the attention maps for each key component:

1. SSFA (Sparse Shape Feature Adaptation). We visualize the attention scores between Entity-Contour-Map tokens and noisy-image tokens. High scores appear for tokens at identical spatial indices, confirming spatial alignment under SSFA.
2. AIA (Attribute Isolation Attention Mask). Using the "three animals (Lego-style lion, ice-style penguin, flame-style eagle)" example mentioned in Q3, we show that at layer 20 the cross-entity attention values drop dramatically once AIA is enabled, directly evidencing attribute leakage suppression.
3. Training-based vs. Training-free AIA. For the Figure 3 column 3 case, layer-39 attention maps reveal that the training-based AIA assigns markedly higher attention scores between the landmark and its water reflection than the training-free variant, evidencing restored global coherence in deeper layers.

Q6: Different key components improve performance in different aspects and the final determined version did not achieve the best results in most of the indicators. Authors should clarify their reasons and dicuss the balance and interply of different components.

Thank you for highlighting the sub-optimal metrics in the final configuration. In the final version, we will incorporate the following explicit justifications for each component:

• SSFA yields a substantial +4.7% gain in class-agnostic mIoU (shown in Table 4).
• AIA improves the average of four region-wise metrics by 1.4 % in Table 4 and effectively prevent attribute leakage. We will add the four new cases mentioned in Q3 to make this benefit concrete.
• Training-based AIA is essential for recovering global coherence, as evidenced by realistic reflections and shadows.
• CITF is retained solely for efficiency. Standard metrics (FID, CLIP, IR, Pick) imperfectly capture perceptual quality. We will provide extra visual results confirming CITF leaves image fidelity almost unchanged.