FOCUS: Unified Vision-Language Modeling for Interactive Editing Driven by Referential Segmentation

Thank you for your review and your positive feedback on our work. Your questions regarding the definition of "unified" and the attribution of performance improvement are very insightful.

Regard weekness1&queston about "unified"

Previous approaches typically employ cascaded pipelines (e.g., LLM+SAM→segmentation→independent editing), where each module is pre-trained independently and lacks joint optimization. This separation often results in suboptimal editing, as the segmentation output may not be well-suited for the requirements of the generation module. In contrast, our approach is end-to-end trainable, enabling joint learning of instruction comprehension, object segmentation, and diffusion-based editing. All components share visual representations and operate within a unified, language-aligned space, which facilitates fine-grained, instruction-driven editing. Compared to Transfusion，which also aims to unify perception and generation，our model achieves joint optimization across all modules, further improving pixel-level accuracy and consistency.

As demonstrated in Table 1, our method surpasses all baselines on the Reason-Edit task (PSNR, SSIM, CLIP Score), showing both precise region localization and strong background preservation under complex instructions. Table 2 and Table 3 further highlight our model’s superior performance in text-guided video (DAVIS) and image (COCO-caption) editing.

Table 1: Quantitative comparison of different methods on understanding and reasoning scenarios. Higher is better for all metrics.

Table 2: Quantitative comparison of different methods on video editing benchmarks. Higher is better for all metrics.

Table 3: Quantitative comparison of different methods on object and background editing. Higher CLIP and lower FID indicate better performance.

Regard weekness2 (Source of performance improvement)

The performance improvements of our model are mainly due to its unified framework design, rather than simply stacking individual modules. To substantiate this, we performed a series of ablation studies to clarify the unique contributions of each encoder module. The results (as shown in the table) are summarized as follows:

• The Vanilla Encoder is essential for vision-language understanding. Removing this module (i.e., using only the Hierarchical + Pixel Encoder) results in a substantial decrease in language comprehension.
• The Hierarchical Encoder offers robust multi-scale visual perception. Excluding this module (i.e., Vanilla + Pixel Encoder) leads to a notable reduction in segmentation accuracy on high-resolution images.
• The Pixel Encoder significantly enhances pixel-level perception for generation tasks. Without it (i.e., Vanilla + Hierarchical Encoder), while segmentation remains strong, the fidelity and detail of generated images are diminished.

These comparisons indicate that any single or dual-encoder combination is inadequate for handling all tasks and presents clear limitations. This demonstrates that only by jointly training these three complementary modules within our unified framework can we fully leverage their synergistic effects, ultimately achieving comprehensive and superior performance.

Table 4: Ablation study results for different encoder combinations on Refcoco, Emu Edit, and Image Understanding benchmarks. Higher is better for all metrics.

Regarding Weakness 3 (Architectural design and autoregressive solutions)

FOCUS is a unified framework that seamlessly integrates multiple components to enable high-quality understanding, segmentation, and image editing in a unified manner—an achievement that remains challenging for previous methods. At its core, our design employs a multimodal autoregressive large language model (MLLM) for instruction reasoning and discrete visual token generation. The diffusion module is incorporated not as a replacement for the autoregressive process, but as an advanced image decoder to enhance output fidelity. Leveraging diffusion allows our model to recover fine-grained details, reduce visual artifacts, and achieve greater robustness compared to direct decoding from discrete tokens. Additionally, the diffusion decoder supports resolution upscaling during generation (e.g., from 256×256 to 512×512), which is crucial for producing high-quality results. Given current technical constraints, the combination of an autoregressive MLLM with a diffusion-based decoder strikes an effective balance among semantic understanding, editing accuracy, and visual quality. We also acknowledge the promise of fully autoregressive generation models and view them as a valuable direction for future research.

Regarding Weakness 4 (Clarity of Prompt and figures)

We have provided detailed clarification in the supplementary material. FOCUS adopts a structured prompt schema (as shown in Table 10) consisting of two components: a task instruction prompt and a condition prompt.

The task instruction prompt defines the model's objective in natural language. For example, “segment the two skiers and perform editing” falls under this category. In contrast, the texts and visual inputs in Figures (a), (b), (c), and (d)—such as scribbles, bounding boxes, and masks—serve as condition prompts, providing necessary contextual information for the task.

In our framework, textual commands like “replace,” “segment,” “remove,” and “modify” are treated as text prompts, while bounding boxes, points, masks, and scribbles are treated as visual prompts, primarily used in interactive and video segmentation tasks.

Technically, we sample CLIP visual features $f_v$​ from the VLLM according to the region coordinates and apply adaptive average pooling to generate final reference features for each visual prompt. Since these visual prompts are also encoded into feature vectors, text and visual prompts can co-exist and jointly guide the segmentation and generation process.

We will revise the figures and descriptions in the final version to ensure that the distinctions between instruction prompts, condition inputs, and model-internal representations are clearly conveyed and easy to understand.

Regarding the Limitations Section:

 Thank you for your feedback. We apologize that the 'Limitations' section, though mentioned in our submission checklist, was omitted from the main manuscript. We will add the limitation: (1) the limited precision when editing small objects due to latent space downsampling; (2) the high computational cost, which challenges replication; and (3) inherited weaknesses from SDXL, such as rendering complex spatial relationships, which we frame as a direction for future work.

Thank you very much for your thorough review and for raising your score to Accept. We greatly appreciate your understanding of our core contribution regarding the unified architecture design.

We will carefully incorporate all the concerns and experimental suggestions you raised throughout the review process into the final manuscript, not limited to Figure 1. Your constructive feedback has been invaluable in improving the quality of our work.

Thank you again for your time and detailed evaluation.

Thank you for your insightful and challenging questions. They have prompted us to reflect more deeply on the core design of our work.

Regarding Weakness 1 & Question 2 (Why an LLM is necessary, instead of directly using a mask to guide the diffusion model):

A basic model that does not incorporate a large language model (LLM)—for example, one that simply uses a user-provided mask to guide a diffusion model—can only handle straightforward image editing tasks. However, such a model is fundamentally unable to process complex natural language instructions that require semantic comprehension and reasoning.

Consider the following examples:

• "Please change the hair of the smiling lady on the left to blonde."
• "Remove the cup in front of the cat on the table."

These instructions involve intricate spatial and attribute-based descriptions such as “on the left,” “smiling,” and “in front of,” which cannot be accurately interpreted or localized using manual masks or conventional vision modules. In our framework, the LLM acts as a crucial bridge, connecting high-level language understanding with precise, pixel-level localization.

FOCUS harnesses the reasoning power of the LLM to accurately identify target regions and execute fine-grained edits in response to complex user instructions. This capability is essential for enabling truly interactive image editing. We will emphasize this point more clearly in the introduction and method sections to further distinguish FOCUS from traditional mask-guided editing approaches.

As demonstrated in Table 1, FOCUS surpasses all baselines in the Reason-Edit benchmark (PSNR, SSIM, CLIP Score), highlighting its ability to accurately localize edits while preserving the background under complex instructions.
Additionally, Tables 2 and 3 present strong results on text-guided video editing (DAVIS) and image editing (COCO-caption), respectively.

Importantly, FOCUS is built upon the relatively lightweight Qwen2.5-3B model. Despite having significantly fewer parameters than mainstream 7B-scale LLMs, it achieves comparable or even superior performance to larger models such as Janus-Pro-7B across multiple multimodal understanding tasks. This design also endows the model with excellent scalability.

 

Table 1: Comparison of different methods on understanding and reasoning scenarios. Higher is better for PSNR, SSIM, and CLIP Score.

Table 2: Comparison of different methods on text-guided video editing (DAVIS). Higher is better.

Table 3: Comparison of different methods on text-guided image editing (COCO-caption). Higher is better for CLIP, lower is better for FID.

 

Regarding Weakness 2 & Question 1 (The necessity and novelty of the dual-branch encoder): Although the dual-branch architecture itself is not novel, its application in our work is essential for addressing the dual requirements of referential editing: LLM-based semantic reasoning and pixel-level accurate segmentation. Our approach enables semantically-aware and pixel-precise editing. Specifically, the global branch is responsible for understanding "what" to edit (i.e., comprehending the language prompt), while the local branch determines "where" to edit (i.e., generating precise masks). Our main contribution lies in the effective integration of these two branches, resulting in a robust end-to-end solution for this challenging task.

The performance improvements of our model are primarily due to its unified framework design, rather than simply stacking individual modules. To support this, we conducted a series of ablation studies to clarify the unique contributions of each encoder module. The results (as shown in the table) are summarized as follows:

• The Vanilla Encoder is essential for vision-language understanding. Removing this module (i.e., using only the Hierarchical + Pixel Encoder) results in a significant decrease in language comprehension.
• The Hierarchical Encoder provides strong multi-scale visual perception. Without this module (i.e., Vanilla + Pixel Encoder), the model exhibits a notable drop in segmentation accuracy on high-resolution images.
• The Pixel Encoder enhances pixel-level perception for generation tasks. Excluding it (i.e., Vanilla + Hierarchical Encoder) maintains robust segmentation, but the detail and quality of generated images are diminished.

These comparisons demonstrate that any single or dual-encoder combination is insufficient to address all aspects of the task, each showing clear limitations. Only by jointly training these three complementary modules within our unified framework can we fully leverage their synergistic effects, ultimately achieving comprehensive and superior performance.

Table 4: Ablation study of different encoder combinations on Refcoco, Emu Edit, and Image Understanding benchmarks. Higher is better.

Regarding Weakness 3 (Table readability): We will add arrow symbols (↑/↓) to all performance tables to clearly indicate which direction represents better performance. In addition, we will provide a detailed analysis in the appendix explaining why using a mask is more effective than using mask token embeddings.

Regarding Weakness 4 (Mask token analysis):

 Our hypothesis is that mask tokens lose precise spatial topological structure after quantization. In contrast, a downsampled binary mask, despite its lower resolution, retains an explicit and continuous spatial guidance signal, which is more friendly to the attention mechanism of the diffusion model.

Table 5: Comparison of mask token and mask usage effects on the diffusion model in EmuEdit. For DINO, CLIP-I, and CLIP-T, higher values indicate better performance; for CLIP-DIR, lower values are better.

Dear Reviewer, Thank you for your review and the insightful questions you've raised about our work. Your concerns have helped us to more clearly position the value of our contributions.

Regarding Weakness 1 & Question 2 (Dependence on manually annotated segmentation data and OOD performance)

Our method should be extensible and employs a highly scalable strategy that eliminates the need for manual annotation by utilizing an automated data generation pipeline. This pipeline combines GroundDINO for object localization, SAM2 for precise segmentation, and a Large Vision-Language Model (LVLM) for further refinement, enabling the efficient production of high-quality training data.

Meanwhile, our model (FOCUS) achieves state-of-the-art (SOTA) performance on out-of-distribution (OOD) benchmarks, including ADE-OV, Citys-OV, PC59-OV, and PAS20-OV. This outstanding performance is largely due to the integrated LLM’s advanced language understanding capabilities for visual grounding, which enable robust results even when processing unfamiliar instructions across a wide range of scenarios.

Table 1: Comparison of segmentation performance on out-of-distribution (OOD) benchmarks.

Regarding Weakness 2 Lack of ablation studies

• The effect of multi-stage training:

  The following table shows the results of using different image resolutions (256, 512, and 1024) for all-stage task training. We observe a clear trend: higher image resolutions improve the model’s performance in image generation and segmentation, but lead to a decline in language understanding. Our analysis suggests that this decline is mainly due to an increase in hallucination issues at higher resolutions—when processing high-resolution images, the model is more likely to generate text that is inconsistent with the visual input. This increased hallucination negatively impacts the model’s language comprehension, highlighting the challenge of balancing visual precision and robust language understanding in multimodal models.

  To address this issue, our multi-stage training strategy starts with simpler tasks at lower image resolutions and gradually increases both the image size and task complexity in subsequent stages. This progressive approach enables our model to achieve strong and balanced performance across all tasks.

  Table 2: The effect of different image resolutions and multi-stage training strategy on model performance across various tasks. ↑ indicates higher is better, ↓ indicates lower is better.

  Image Size	MJHQ30K	GenAI-Bench		Image Understanding			RefCOCO
  	FID(↓)	Basic(↑)	Adv.(↑)	POPE(↑)	MMB(↑)	SEED(↑)	testA(↑)	TestB(↑)	Valid(↑)
  256	11.85	0.63	0.56	85.3	73.1	70.1	78.5	79.2	75.4
  512	7.39	0.70	0.61	87.3	80.6	71.3	81.3	82.4	79.8
  1024	6.03	0.76	0.69	77.6(↓)	68.2(↓)	68.8(↓)	83.4	84.9	81.5
  256->1024(Our)	6.05	0.83	0.72	88.0	81.5	73.9	84.1	86.3	82.7

• The effect of multi-scale features in Gated Cross-Attention.

  To effectively fuse multi-scale features, our model employs a Gated Cross-Attention module to process outputs from the ConvNeXt-L backbone, achieving a balance between fine-grained detail and global contextual understanding. Specifically, we design a 3-layer Gated Cross-Attention adapter that hierarchically integrates three distinct feature scales: the first layer focuses on fine features ($f_2$), the second on mid-level features ($f_3$), and the third on coarse semantic features ($f_4$).

  As illustrated in the table, we compare our three-scale model ($f_2$, $f_3$, $f_4$) with variants that utilize only a single scale ($f_4$), dual scales ($f_3$, $f_4$), and all four scales ($f_2$–$f_4$). The results (mIoU on RefCOCO) demonstrate that our multi-scale fusion strategy is essential for accurately segmenting fine details, providing the optimal balance between performance and computational efficiency.

  Table 3: Ablation study on the effect of multi-scale feature fusion in Gated Cross-Attention.

  Model Variant	#Scales	RefCOCO mIoU(↑)	MMBench Acc.(↑)	CLIP-T(Edit)(↑)
  FOCUS-SingleScale (Fine)	1	79.5	72.8	0.265
  FOCUS-SingleScale (Coarse)	1	78.2	73.9	0.260
  FOCUS-DualScale	2	80.5	73.5	0.270
  FOCUS (Ours)	3	81.4	73.9	0.275

Regarding Weakness 3 (Comparison with works like NExT-GPT, VisionLLM, etc.):

Previous frameworks such as NExT-GPT and VisionLLM v1 & v2 have made progress toward unifying vision tasks, but their reliance primarily on text conditioning limits their ability to achieve precise spatial control in image editing. In contrast, our method stands out for its pixel-aware design: by utilizing the edge information of the target region, our approach enables highly accurate and localized modifications.

This advantage is consistently reflected across multiple benchmarks. Tables 6 and 7 showcase our superior performance in text-guided video editing (DAVIS) and image editing (COCO-caption), respectively. Furthermore, as illustrated in Table 8, our model achieves segmentation accuracy on par with state-of-the-art (SOTA) methods such as VisionLLMv2, while uniquely offering robust editing and generation capabilities.

We also recognize Bagel, a concurrent work with promising results, as an important reference point. We believe that future progress can be achieved by drawing on the strengths and insights of both approaches.

Table 6: Quantitative comparison of text-guided video editing methods on DAVIS.

Table 7: Quantitative comparison of text-guided image editing methods on COCO-caption.

Table 8: Quantitative comparison of segmentation performance on RefCOCO and ReasonSeg benchmarks.

Regarding Question 1 the role of the segmentation mask

To clearly illustrate the fundamental importance of the segmentation mask in our framework, we have designed a series of comparative experiments:

1. FOCUS (Full Model): Utilizes internally generated masks that are precisely aligned with the given language instructions.
2. Baseline 1 (No Mask Guidance): Removes mask guidance entirely, relying solely on text instructions to guide the diffusion model during editing.
3. Baseline 2 (General-Purpose Mask): Employs masks produced by an external, pre-trained, general-purpose segmentation model (Grounding+SAM2) for guidance.

These results demonstrate that precise and instruction-aligned segmentation masks are essential for optimal editing performance in our framework.

Table 9: Ablation study on the effect of segmentation mask guidance on EmuEdit benchmark.

Dear reviewer,

The author has responded to your question. As the discussion period is nearing its end, please review their reply to see if it addresses your concerns or if you have any follow-up questions. Kindly remember to submit your final review by the deadline.

Area Chair

Regarding Weakness 1 & Question3 (Lack of ablation studies):

• The effect of multi-stage training:

  The following table shows the results of using different image resolutions (256, 512, and 1024) for all-stage task training. We observe a clear trend: higher image resolutions improve the model’s performance in image generation and segmentation, but lead to a decline in language understanding. Our analysis suggests that this decline is mainly due to an increase in hallucination issues at higher resolutions—when processing high-resolution images, the model is more likely to generate text that is inconsistent with the visual input. This increased hallucination negatively impacts the model’s language comprehension, highlighting the challenge of balancing visual precision and robust language understanding in multimodal models.

  To address this issue, our multi-stage training strategy starts with simpler tasks at lower image resolutions and gradually increases both the image size and task complexity in subsequent stages. This progressive approach enables our model to achieve strong and balanced performance across all tasks.

  Table 1: The effect of different image resolutions and multi-stage training strategy on model performance across various tasks. ↑ indicates higher is better, ↓ indicates lower is better.

  Image Size	MJHQ30K	GenAI-Bench		Image Understanding			RefCOCO
  	FID(↓)	Basic(↑)	Adv.(↑)	POPE(↑)	MMB(↑)	SEED(↑)	testA(↑)	TestB(↑)	Valid(↑)
  256	11.85	0.63	0.56	85.3	73.1	70.1	78.5	79.2	75.4
  512	7.39	0.70	0.61	87.3	80.6	71.3	81.3	82.4	79.8
  1024	6.03	0.76	0.69	77.6(↓)	68.2(↓)	68.8(↓)	83.4	84.9	81.5
  256->1024(Our)	6.05	0.83	0.72	88.0	81.5	73.9	84.1	86.3	82.7

• The effect of multi-scale features in Gated Cross-Attention.

  To effectively fuse multi-scale features, our model employs a Gated Cross-Attention module to process outputs from the ConvNeXt-L backbone, achieving a balance between fine-grained detail and global contextual understanding. Specifically, we design a 3-layer Gated Cross-Attention adapter that hierarchically integrates three distinct feature scales: the first layer focuses on fine features ($f_2$), the second on mid-level features ($f_3$), and the third on coarse semantic features ($f_4$).

  As illustrated in the table, we compare our three-scale model ($f_2$, $f_3$, $f_4$) with variants that utilize only a single scale ($f_4$), dual scales ($f_3$, $f_4$), and all four scales ($f_2$–$f_4$). The results (mIoU on RefCOCO) demonstrate that our multi-scale fusion strategy is essential for accurately segmenting fine details, providing the optimal balance between performance and computational efficiency.

  Table 2: Ablation study on the effect of multi-scale feature fusion in Gated Cross-Attention.

  Model Variant	#Scales	RefCOCO mIoU(↑)	MMBench Acc.(↑)	CLIP-T(Edit)(↑)
  FOCUS-SingleScale (Fine)	1	79.5	72.8	0.265
  FOCUS-SingleScale (Coarse)	1	78.2	73.9	0.260
  FOCUS-DualScale	2	80.5	73.5	0.270
  FOCUS (Ours)	3	81.4	73.9	0.275

• The effect of continuous visual tokens. Our ablation study demonstrates that feeding continuous visual features directly into the LLM is essential for optimal performance. By bypassing discrete tokenization, this approach effectively reduces information loss and retains fine-grained visual details. As a result, the LLM receives a richer and more precise visual representation, which significantly enhances its ability to comprehend complex multimodal scenarios and accurately interpret referential instructions.

  Table 3: Ablation study on the effect of continuous visual token input on model performance.

  Continuous Input	MJHQ30K	GenAI-Bench		Image Understanding
  	FID(↓)	Basic(↑)	Adv.(↑)	POPE(↑)	MMB(↑)	SEED(↑)
  ×	6.85	0.68	0.65	82.1	50.4	56.0
  √	6.05	0.83	0.72	85.3	70.9	66.6

• The effect of LLM decoding constraints: The results in the table demonstrate the effectiveness of our decoding constraint, which requires the model to first identify object categories before generating segmentation masks. This structured decoding process not only enhances semantic consistency and reduces the risk of arbitrary segmentation from ambiguous instructions, but also leads to a significant improvement in final segmentation accuracy.

  Table 4: Ablation study on the effect of LLM decoding constraints on segmentation performance.

  constraints	YT-VIS (mAP)	RefCOCO (cIoU)
  ×	50.8	83.5
  √	54.6	84.1

Regarding Weakness 2 & Question2 (Lack of Limitations Discussion): Thank you for your feedback. We apologize that the 'Limitations' section, though mentioned in our submission checklist, was omitted from the main manuscript. We will add the limitation: (1) the limited precision when editing small objects due to latent space downsampling; (2) the high computational cost, which challenges replication; and (3) inherited weaknesses from SDXL, such as rendering complex spatial relationships, which we frame as a direction for future work.

Regarding Weakness 3&4 (Writing Issues in the Experimental Section): Thank you for your suggestions and feedback. We will revise the manuscript to further improve the clarity and completeness of the paper. First, we will add citations in the main paper for all datasets (e.g., POPE, MMBench) and evaluation metrics (e.g., FID, MJHQ) to ensure proper attribution. Second, we have corrected the incorrect bolding in Table 2 and added appropriate bolding in Table 3 to accurately highlight the best-performing results. Finally, we will move Figures 1 and 2 from the supplementary material into the main paper to enhance clarity and readability.

Regarding Question1:

The progressive training paradigm we adopt, by elevating task complexity and resolution in stages, has at its core the optimization of data utilization strategy and effectiveness, rather than a reliance on the expansion of data scale.