We sincerely thank the reviewer for the professional, thoughtful and constructive comments!

Weaknesses

W.1 Missing Some Related Works

We appreciate the reviewer’s professional suggestion to strengthen the contextual grounding of our contributions. We will discuss and analyze these works in the revised paper.

W.2 Not Good Performance on Certain Tasks

Thank you for your insightful comment! We respectfully believe the concern is not due to limitations in model's performance, but rather due to the lack of a standardized output representation that is both aligned with the diffusion model's predictions and suitable for established evaluation metrics. Owing to the significant variation in representations among different tasks, existing methods typically employ task-specific decoders, especially for instance-level tasks such as pose estimation.

Unlike prior works that often omit quantitative evaluation on these hard-to-evaluate perceptual tasks, our paper provides usable quantitative metrics. We believe this is a critical issue that cannot be overlooked. As diffusion models for perception continue to develop, establishing ways to compute such metrics will be essential for fair and rigorous comparison across methods. Relying only on easy-to-evaluate tasks such as depth or surface normal estimation is insufficient to fully distinguish the performance of different approaches. Although our current metrics—affected by significant post-processing errors—may appear to underperform compared to traditional task-specific methods, we believe that they can still serve as a practical and necessary evaluation baseline for future diffusion-based perceptual models.

Furthermore, we believe these challenges can be addressed. For instance, by representing poses with heatmaps, although the errors in post-processing still exists, we achieve notably higher evaluation metrics. Furthermore, we think that a more fundamental solution may lies in exploring the VAE decoder. Existing work[1] shows fine-tuning VAE decoders has yielded higher-quality point maps. We believe this is insightful and promising.

[1] GeometryCrafter: Consistent Geometry Estimation for Open-world Videos with Diffusion Priors

W.3 Key challenges in Diffusion Model for Perception

Thank you for your professional comments! In our opinion, inference speed is one of the critical problems when applying diffusion models to perception tasks, due to the typical requirement for many iterative steps. Although previous works have achieved 1-step inference in single-task settings, this problem is still under-explored in multi-task scenarios.

We explore the feasibility of few-step even 1-step inference and find that our model demonstrates high-quality results in few-step inference settings without significant performance drop. Different tasks exhibit varying sensitivities to the reduction of inference steps. For instance, depth and normal estimation can be performed with as few as one inference step without significant performance degradation. For more intricate tasks like interactive segmentation, while a single step leads to degradation, a moderate increase to 7 inference steps still yields results with minimal performance compromise, which is a 4x acceleration. To the best of our knowledge, this is the first time such a capability is demonstrated in diffusion model for multi-task perception. It strongly supports the advantage of flow-matching-based diffusion models in solving perception tasks.

We believe this is because flow matching explicitly imposes linear constraints on intermediate noisy latents, enforcing them to be linear interpolations between noise and the target. As a result, the denoising trajectory is encouraged to be as straight as possible. Moreover, low-level perception tasks typically do not require conditioning on complex prompts. Together, these factors enable the model to produce accurate outputs even with a reduced number of inference steps, as the inference trajectory remains well-aligned with the learned straight path.

We also apologize for the confusing presentation in Section 4.3. In Section 4.3 of our paper, we follow the GenPercept training strategy, where the model is directly trained to predict the target from pure noise in a single step. This approach discards the linear constraints on intermediate noisy latents that are explicitly enforced by flow matching, thus producing poor results. Here we show that when the model is trained under a multi-step flow matching denoising setting, it can naturally support few-step inference. We will revise this part of the paper accordingly.

In summary, our paper contributes new knowledge to the community by showing that flow-matching-based diffusion models, when trained with multi-step denoising, inherently support efficient few-step and even one-step inference in perception tasks. We kindly refer you to our response to Reviewer MKSc, where we provide more comprehensive analysis and detailed experimental results.

W.4 Dangerous Expressions

Thank you for pointing this out. What we intended to convey is that, our approach can achieve competitive performance using substantially less training data compared to specialized expert models (our 600K images vs. 1B for SAM, 500K vs. 62M samples for DepthAnything2). We will revise the wording for better clarity.

W.5 Referring Segmentation

Thank you for your professional suggestions! We conduct additional referring segmentation experiments using RefCOCOg and demonstrate our method can show good performance. We will also supplement qualitative visualizations in the revised version of the paper, as images cannot be provided in response.

Questions

Q.1 Emergent Behaviors and Task Composition

In our view, emergence behavior typically requires training on extremely large-scale data. The amount of data used in our work is still far from that scale. Nevertheless, we do observe early signs of such potential in our model. Our model does exhibit certain compositional capability. Here are some cases:

• When composing segmentation and human pose captions, the model sometimes outputs results with both segmentation masks and pose keypoints. Similar behavior is also observed in depth and surface normal estimation. The model can predict depth or normal of the object labelled by point inputs and output pure black mask for other regions, but with a lower success rate.
• In entity segmentation, we can provide points as additional input to guide finer-grained predictions. E.g., a bookshelf is segmented as a single entity, but with additional points, the model can further segment the individual books on the shelf. To summarize, while the model was not trained on combined entity and interactive segmentation data, their integration is achievable at inference through prompt composition.

We will further provide qualitative visualizations above in the revised paper, as images cannot be provided during response.

As for sequentially conducting multiple tasks one by one, this is fully available. Furthermore, thanks to our model’s few-step inference capability, this process remains highly efficient.

Discussion

D.1 High-level Tasks and Emergent Behaviors

Tackling complex multi-modal perceptual tasks and exploring "emergent behaviors" are very attractive. In our opinion, achieving these goals requires great understanding capability. In this work, we demonstrate that generative models can be effectively leveraged to build a unified model for multiple perceptual tasks. Recently, a new model which exhibits strong capabilities in both generation and understanding, BAGEL[2], has drawn our attention. It outperforms previous diffusion and vision-language models in both generation and comprehension. We believe this model is promising to solve high-level multi-modal perception tasks. We have made attempts on BAGEL our next step and are currently exploring it. And we are confident this will reveal novel and compelling features on complex, high-level tasks, even emergent behaviors.

[2] BAGEL: Emerging Properties in Unified Multimodal Pretraining

We sincerely thank you again for your professional review, constructive comments, and insightful discussion! Please let us know if you have any further concerns. We also look forward to further constructive discussions with you! :)

We certainly will! We sincerely thank you again for your professional and constructive feedback. We are very happy and grateful for the opportunity to engage in such a meaningful discussion!

Thank you for your positive and constructive comments and insightful suggestions on our paper. We address the comments and questions as follows:

W.1 High-level Task

Thank you for your feedback. We have conducted additional experiments to evaluate our model on the RefCOCOg dataset for referring segmentation. The results demonstrate that our approach is effective for high-level visual tasks. Unfortunately image submission in the rebuttal is not allowed. We will also include more qualitative visualizations in the revised version of the paper.

W.2 Inference Time

We sincerely thank you for the insightful comment.

We Naturally Support Few-step Inference

We conduct experiments and observe that our model inherently supports few-step inference for perception tasks without any additional techniques, and shows very little performance degradation. The effectiveness of few-step acceleration varies across different tasks. For tasks such as depth and surface normal estimation, the number of inference steps can be reduced to as few as one with acceptable slight performance degradation. For more complex tasks such as interactive segmentation, the model is still able to achieve comparable results using significantly fewer steps while maintaining competitive performance, for example, 7 step, which is still a 4x acceleration, as demonstrated below:

Analysis

We believe this is because flow matching explicitly imposes linear constraints at each intermediate denoising step—specifically, each noisy latent is constructed as a linear interpolation between the pure noise and the target signal. This process effectively straightens the denoising trajectory, allowing the model to follow an approximately linear path even when using only a few inference steps. In contrast, if the model is trained solely with one-step denoising, the intermediate steps are not constrained and lacks this linear constraint across the trajectory, thus producing poor results as we show in Section 4.3. In contrast, traditional ODE-based diffusion models do not impose such linear trajectory constraints, and therefore cannot support inference with few denoising steps (such as 4 steps) after being trained with multi-step denoising (such as 50 steps).

Our additional experiment proves this. We further experiment with PixArt-α, which uses a DiT-style architecture but adopts a standard ODE-based scheduler. Its results significantly deteriorate when the number of inference steps is reduced, further supporting our analysis above.

In image generation tasks, simply reducing inference steps in a flow-matching-based text-to-image model also leads to noticeable quality degradation. This is due to the high complexity and variability introduced by diverse text prompts. In contrast, our perception tasks eliminate the influence of textual prompts, which we believe explains why prior works like One Diffusion require 50~100 inference steps for denoising while ours works well with just a few steps.

For comparisons on inference efficiency, we select One Diffusion as baseline and conduct a comparative study on our shared task—depth estimation—under varying numbers of inference steps. Unlike One Diffusion, which suffers from significant performance degradation during few-step inference and fails to produce reasonable results in the 1-step setting, our method is capable of generating high-quality outputs even with just a single inference step. The results demonstrate that our method significantly outperforms One Diffusion in both efficiency and output quality.

Revise the according parts in Section 4.3

We apologize for the confusing presentation in Section 4.3. In Section 4.3, we say one-step inference is ineffective is because those results are obtained with the model trained to predict the target directly from noise in a single forward pass, which follows the setting of previous approach Genpercept. Here we show that when the model is trained under a multi-step flow matching denoising setting, it can naturally support few-step inference. In the revised version of the paper, we will clarify this point and provide a more complete analysis by incorporating additional results that demonstrate the few-step inference capability enabled by multi-step flow matching training.

Once again, we sincerely express our sincere gratitude for your insightful and professional comments! Please let us know if you have any further concerns.

We sincerely thank the reviewer for the constructive feedback.

Weaknesses

W.1 Incremental improvement over One Diffusion

We respectfully disagree with the claim that our work is merely an incremental improvement over One Diffusion. The most fundamental difference lies in the target we want to address. We focus only on perception tasks rather than generation tasks in One Diffusion, which significantly leads to different research goals and conclusions. Specifically, our valuable findings and conclusions, such as suitable diffusion architectures for multiple perception tasks, input strategies, role of CFG, and our inherent support for few-step inference are all towards better repurposing diffusion models to a generalist perception model. Especially our method's direct support for few-step inference shows a salient distinction to One Diffusion.

Key Distinction and Innovation: Enabling Few-Step Inference for Multi-Task Perception with Diffusion Models

To further highlight the advantages and contributions of our approach compared to One Diffusion, we investigate the feasibility of few-step, even 1-step inference of our method. Our findings reveal that the proposed model achieves high-quality results under significantly reduced inference steps, with minimal performance degradation. To the best of our knowledge, this is the first time such a capability is demonstrated in a diffusion-based multi-task perception model, providing strong evidence for the effectiveness in efficient inference for perceptual tasks. This highlights our contributions.

We believe the superior few-step inference capability of our model stems from the intrinsic properties of flow matching, which enforces linear constraints at every intermediate denoising step. In particular, each noisy latent is explicitly constructed as a linear interpolation between pure noise and the target signal. This linearity effectively regularizes the denoising trajectory, encouraging the model to follow a near-linear path in latent space, which remains stable even under a reduced number of inference steps.

We also apologize for the unclear explanation in Section 4.3. The failure of one-step inference reported in that section stems from the fact that the model was trained to predict the target directly from pure noise in a single step, without involving intermediate latent states, following the strategy of GenPercept. In contrast, here we demonstrate that when the model is trained using a multi-step denoising process with intermediate noisy latents, it can naturally support few-step inference without additional techniques. We will revise the paper for better clarity.

In contrast, One Diffusion requires 50 to 100 inference steps to produce satisfactory results, while our model maintains high performance with significantly fewer steps. We attribute this discrepancy to One Diffusion’s strong reliance on detailed text prompts, which introduces substantial variability into the denoising process and increases the complexity of the underlying data distribution that the model must learn. Similarly, in flow-matching-based text-to-image generation models, aggressively reducing the number of inference steps often leads to notable quality degradation. These observations both suggest that the presence of complex textual prompts complicates the generative process, necessitating more denoising steps. This, in turn, supports the design choice in our work to eliminate complex textual conditioning in perception tasks, thereby simplifying the target distribution and enabling efficient and effective few-step inference.

We provide additional few-step inference results across different tasks. Our method effectively reduces the number of inference steps while incurring only minimal performance degradation. For complex tasks such as interactive segmentation, although 1-step inference remains infeasible, our method reduces the required steps to 7, achieving a 4× speed-up compared to the original setting while preserving strong performance.

In summary, as acknowledged by other reviewers, our work goes well beyond simple incrementation. It provides a thorough and systematic exploration of diffusion models in the context of multi-task visual perception, establishing both novel capabilities and practical advantages.

W.2 On computational cost

We appreciate the concern regarding computational overhead. However, we would like to clarify that our training cost is acceptable compared to existing efforts:

• SAM 256 A100 GPUs
• One Diffusion 64 H100 GPUs
• ours 4 H100 GPUs

We believe this makes our setup significantly more accessible and reproducible for the community. As for inference efficiency, we further show that our model supports few-step inference without significant degradation in performance, substantially reducing runtime cost at test time. Furthermore, we will release our model to facilitate future research and development.

W.3 Quantitative Evaluation on Few-shot Training

Thank you for the valuable suggestion. We have applied the same set of 50 images for fine-tuning across ours and baseline methods. The results demonstrate the effectiveness of our approach in few-shot scenarios.

Questions

Q.1 On Prompt Generation Using Image Captioning Models

We acknowledge the use of image captioning models as a possible solution of task prompts. However, these captions:

• Ultimately still originate from human knowledge.
• Different models and hyperparameters may yield different prompts for the same image.
• Captioning models have hallucination problems, a well-known and unresolved issue.
• Generating high-quality captions typically requires large models, which introduces significant additional computational cost.
• Fail to reduce inference step as we show in W.1. Even more steps (50-100) are required.

Therefore, relying on caption-generated prompts does not fundamentally solve the problem. For language-agnostic perception tasks, we argue that introducing unnecessary language components is neither essential nor clearly beneficial. Additionally, as we mentioned, our method demonstrates strong few-step inference capabilities, whereas One Diffusion sometimes requires up to 100 steps to produce results. We believe this is due to the complex prompts it needs, which necessitates more timesteps to arrive at a reasonable prediction.

Q.2 Clarification on Adaptation Experiment and "Pixel-aligned Training"

We apologize for the lack of clarity. This experiment involves using SD3 and Diception as base models, and applying LoRA-based fine-tuning on just 50 training images to demonstrate task adaptation.

The term pixel-aligned training refers to training on pixel-aligned perception tasks (e.g., depth, normal, segmentation, pose). After being trained on perceptual tasks, our model demonstrates faster and more effective adaptation to new tasks than SD3. We will clarify this point in the revised manuscript to avoid any potential ambiguity.

Q.3 Motivation for CFG

CFG is a widely used technique in conditional diffusion models for improving image generation quality without additional training. Our motivation is to investigate whether such generation-oriented techniques could also enhance performance when repurposing diffusion models for perception tasks.

We sincerely thank you again for your constructive feedback! Please let us know if you have any further concerns.

Thank you very much for your instantaneous response. I believe all of my concerns have been thoroughly addressed.

Thank you for your constructive response. Some of my concerns have been adequately addressed.

However, the main contribution still appears to be the construction of a framework that applies techniques commonly used in generative models to diffusion models—techniques that have not been explicitly tailored for perception tasks.

Moreover, the current writing of the main paper does not clearly convey the rebuttal’s claim regarding the advantage of few-step inference.

We sincerely thank the reviewer for the valuable feedback. We address the comments and questions as follows:

Weaknesses

W.1 Clarification on Pixel-Aligned Training

We apologize for the confusion. In our paper, the term "pixel-aligned training" refers to our training on pixel-level perceptual tasks demonstrated in our paper (such as depth, surface normal, segmentation, human pose). No additional techniques and strategies are applied. Training on these perceptual tasks effectively facilitates detail preservation when adapt to downstream tasks such as image highlighting. We will clarify this in the revised manuscript to avoid ambiguity.

W.2 Scalability on Larger Datasets

Thank you for your valuable comments. We acknowledge the limitation of not validating the scalability of our model on larger datasets. This is primarily due to constraints in computational resources. Nevertheless, the current results are still very promising and demonstrate the potential of our approach. This is supported by our comprehensive evaluation across a total of 33 widely-used benchmark datasets. We are committed to explore scalability further with more computational resources and larger training dataset in future work.

W.3 Paper Organization

Thank you for pointing this out. In the revised version, we will move some of the related work section to the appendix and highlight our experiment results and analyses in the main paper.

Questions

Q.1 Optimization Schemes and Alternative Solutions for the Errors Introduced by Post-processing

Thank you for your constructive comments. In our paper, we have achieved quantitative metrics with a simple post-processing step, which represents a qualitative leap over prior approaches that relied solely on visualization. We regard evaluating these tasks as a fundamental challenge that must be addressed. However, prior works largely overlook this problem. As diffusion models for perception continue to evolve, relying solely on tasks such as depth and normal estimation offers only a limited view and fails to adequately evaluate the multi-task diffusion models. While our current evaluation metrics may look underperformed to traditional task-specific approaches due to the inherent errors introduced by post-processing, we believe it still provides a preliminary baseline for future diffusion-based perceptual methods.

Given the considerable variation in representations across tasks, especially in instance-level settings such as pose estimation, existing methods commonly resort to individually designed decoders tailored to each task. In the context of Diffusion Model, we believe a suitable solution may lie in further exploration of optimizing the VAE decoder. Recent works [1] have shown that fine-tuning the VAE can achieve high-quality point map estimation. Inspired by these efforts, we believe optimizing VAE for decoding the final mask, and even decoding results across multiple tasks, is promising. We will delve into this problem in detail in our future work.

[1] GeometryCrafter: Consistent Geometry Estimation for Open-world Videos with Diffusion Priors

Additionally, we also provide the results of some simple alternative solutions:

• For pose estimation, we conduct a primary experiment on a more complex heatmap-based keypoint representation.

• For segmentation, we randomly sample points within each RGB mask and re-run the interactive segmentation. This can lead to improvement, but does not address the root problem.

These result shows our method could benefit from improved post-processing. We believe the result still has room for improvement. However, to tackle all instance level perception tasks, we assume the RGB space is good for visualizations, but not efficient as specially designed decoders, which leads to future research questions. In summary, to fundamentally address this problem, we believe the exploration of a general multi-task VAE decoder holds potential.

Q.2 CFG for segmentation

Thank you for pointing this out. We conduct additional experiments and show that CFG has negligible impact on segmentation quality, which is consistent with the visualizations presented in Figure S5 of the supplementary material.

Paper Formatting Concerns

Thank you for the suggestion. We appreciate the feedback and will revise the manuscript to reduce the use of italics and apply emphasis more appropriately to improve readability and presentation.

We sincerely thank you again for your highly valuable and constructive suggestions! Please let us know if you have any further concerns.

Thanks to the authors for their thorough responses to my review comments. The major concerns have been addressed. Therefore, I will upgrade the score to "accept". Hope this job could open up future research.