UFO: A Unified Approach to Fine-grained Visual Perception via Open-ended Language Interface

Dear Reviewer vFSU,

Many thanks for your recognition of our work. We will address your questions one by one below.

[Q1] Have the authors explored using the visual features directly from the Vision Transformer within the MLLM instead?

Thanks for asking about this important design. We have conducted comprehensive experiments to compare the differences between using ViT features and using visual features from the LLM. As shown in the table below, relying on ViT features results in a significant drop in performance. We attribute this to the following reasons:

1. Compared to ViT features, the visual features output by the LLM are better aligned with the feature space of the mask embeddings.
2. The LLM-generated visual features are more discriminative. Because we place the text before the image, the LLM can attend to the text prompt when processing the visual features. As a result, LLM can dynamically emphasize image regions relevant to the prompt, which helps the mask embeddings better localize the target. Reversing the input order (image before text, second row in the table) will also lead to a performance drop, highlighting the importance of contextualizing visual features with textual information.

[Q2] Could the generation of synonyms for COCO categories potentially affect the evaluation metrics? What is the method used to compute the detection score?

In the evaluation, we find that all categories predicted by the model are within the 80 classes provided by COCO, and there are no cases of predicting synonyms of the categories. This is mainly because we specify the category names in the text prompt, and the model is able to follow the prompt well.

We use the softmax logit of the first token predicted in each text sequence as the confidence score for the entire prediction. For example, for the sequence "Duck, <box>...", we use the score associated with the "Duck" token.

[Q3] Whether inferring bounding boxes by computing the minimum enclosing rectangles from the predicted masks is a feasible and effective solution.

Yes, predicting bounding boxes using the mask2box approach is feasible, and we have also included this ablation in Appendix F. The performance of UFO-ViT-B on the COCO Detection using this method is presented in the table below. The performance is slightly lower than predicting the textual box. This is mainly because some low-confidence masks have outlier predictions, leading to distorted boxes. The inference speed is also slower, as the mask sequences are generally longer than the box. Therefore, to achieve better accuracy and efficiency, we retain the method of predicting text coordinates for detection tasks.

---

Thanks for your professional comments, we will add these experiments and details to our revision. Feel free to let us know if you have any further questions or concerns :-).

Dear Reviewer 99r1,

We sincerely appreciate your acknowledgment of the significance, intuition, thorough experiments, and clarity of our paper. We will respond to your questions one by one.

[W1] Explanation of open-ended language interface.

Thanks for pointing out this important concept. In the paper, we refer to the "open-ended language interface" as the capability to output variable-length text sequences, with termination indicated by the <END> token. This stands in contrast to prior works such as GiT [1], which are limited to fixed-length outputs.

Our method does not conflict with open-ended language interfaces.

1. Although we require fixed-length <MASK> tokens for segmentation, we still perform text generation in an open-ended manner. Only after text generation is complete, we check whether the output contains <MASK> segments of the required length.
2. The interaction with image features is also carried out only after the text generation is finished.

Notably, after the open-ended text generation ends, our segmentation mask is determined by the image features and mask embeddings. In contrast, SAM4MLLM and SegAgent can only produce coarse prompts by text generation, which requires further refinement by SAM. We will clarify this term in our revision.

[1] GiT: Towards Generalist Vision Transformer through Universal Language Interface. ECCV 2024

[W2] Details on the inference and evaluation processes for object detection and instance segmentation tasks.

[W2.1] How does the method handle multiple objects and categories?

We handle this by the parallel decoding described in Section 3.4, which is also illustrated in Figure 3. During inference, we sample grid points over the image of size $K\times K$, resulting in a total of $M = K^2$ points. Each grid point serves as a local prompt and is responsible for detecting its nearby object. We interpolate the image features at each grid location to extract the grid features: grid_feats = torch.nn.functional.grid_sample(image_feats, grid_locations)

We then input the text prompt, the image features, and the grid features into the LLM. Take the rightmost image in Figure 3 as an example:

$$\text{Detect cow, person, duck.}<\mathrm{Image}><\mathrm{Grid}_1><\mathrm{Grid}_2>\ldots<\mathrm{Grid}_M>$$

<Grid> refers to the <Local> token in Section 3.4. To ensure independence among $M$ predictions, we adjust the attention mask so that each grid feature is isolated from the others. Then we start generating in an autoregressive manner, with the difference that we predict $M$ tokens at each forward instead of just one.

$$<\mathrm{Grid}_1><\mathrm{Grid}_2>\ldots<\mathrm{Grid}_M> | <T^1_1><T^1_2>\ldots<T^1_M> | <T^2_1><T^2_2>\ldots<T^2_M> | \ldots$$

where $T^j_i$ denotes the $j$-th token in the $i$-th sequence and | distinguishes the tokens of each forward pass. When all $M$ sequences have predicted the <END> token, decoding ends. Finally, we obtain M sequences with different objects and categories, which might look like: (1) Duck, <box>... (2) Background $\cdot \cdot \cdot$ (M) Cow, <box>...

For instance segmentation, the process remains the same except that the textual boxes are replaced with <MASK> tokens.

[W2.2] How many forward passes are required?

As explained above, only $1+L$ forward passes are required, where $L$ is the maximum length of the output sequence. Typically, for detection, $L$ is less than 20, and for segmentation, $L$ is less than 30. Thanks to our parallel decoding strategy, the number of forward passes is independent of the number of objects in the image, which greatly accelerates inference. For example, in the semantic segmentation, UFO-ViT-B achieves 4.8 FPS, which is 3$\times$ faster than SAM-B (1.6 FPS). More speed results are in Appendix Table 9.

[W2.3] Are there any additional post-processing steps?

Yes, the post-processing mainly consists of the following steps:

1. Pattern matching: We need to extract class names and box coordinates from text sequences.
2. Extract the confidence scores. We use the softmax logit of the first token predicted as the confidence score for the prediction. For example, for the sequence "Duck, <box>...", we use the score associated with the "Duck" token.
3. Finally, we apply NMS to filter boxes and masks.

[W3] Some newer and stronger works are not included.

Many thanks for listing these outstanding works. We compare our method with these approaches on the RES and ReasonSeg below. UFO achieves a highly competitive performance on RES, outperforming all methods except HyperSeg on average. On ReasonSeg, UFO achieves the best performance, surpassing HyperSeg by 4.3 gIoU on the validation set and Ground-V by 14.9 gIoU on the test set. These results show that UFO unifies reasoning and segmentation more effectively. We will add these comparisons in the revision.

 

[Q1] Ablation of supervision of image tokens.

Thanks for this insightful question. We explore the impact of applying supervision to image tokens on UFO-InternVL2.5-8B:

As shown in the table, supervising image tokens can largely improve segmentation while maintaining original capabilities. We attribute this to two reasons:

1. InternVL2.5-8B does not explicitly supervise image tokens during training; instead, it relies on sparse textual signals, which leads to insufficient preservation of fine details (e.g., object boundaries) in the visual features. Therefore, directly using these visual features can lead to degraded segmentation performance.
2. The supervision in the segmentation provides fine-grained and direct signals that help preserve richer details in the visual features, thereby enhancing their discriminability.

[Q2] Can the embedding retrieval approach also be applied to detection tasks? For example, obtain box embeddings through interactions with image tokens.

Yes, there are two different implementations:

1. Mask2Box: Convert the mask into a box by extracting its minimum enclosing rectangle. Its performance is slightly lower than predicting textual box. This is mainly because some low-confidence masks have outlier predictions, leading to distorted boxes. The inference speed is also slower, as the mask sequences are generally longer than box.
2. Special Token: Predicting box coordinates by interactions with image features. Specifically, we still decompose the detection task into multiple sub-tasks, each corresponding to a grid point. Each point predicts the textual class and four special tokens (<X1>, <Y1>, <X2>, <Y2>) representing the box offsets relative to the point. Each offset (e.g., $\Delta x_1$) is computed using the scaled dot product between the token embedding $e_{X1}$ and the grid feature $h_g$, followed by a sigmoid:

$ \Delta x_1 &= \sigma\left(\frac{e_{X1}^\top h_g}{\sqrt{d}}\right) $

where $\sigma(\cdot)$ is the sigmoid function and $d$ is the dimension. As grid features are interpolated from image features, this can be seen as extracting box information from the image.

This approach is faster due to the shorter sequence, but its performance is worse than textual boxes. There may be two reasons: 1. Shorter sequences are less expressive. 2. This approach must store all offset information in a single grid feature, while the original method can predict textual boxes by attending to both image and grid features.

---

Thanks again for your professional, detailed, and valuable reviews! We have done our best to address each of your concerns and hope our response can resolve them. Please let us know if you have any other questions. We will actively join the discussion until the end of the rebuttal period. We are looking forward to hearing from you :-)!

Dear Reviewer ZRJs,

Many thanks for recognizing the novelty, versatility, effectiveness, strong performance and scalability of UFO. We address your questions as follows.

[W1] How these specialized tasks might conflict with or affect the model's ability to handle broader, more general Q&A tasks?

Thanks for pointing out this important concern. We have included the performance on general QA in Table 5 of the main paper, and we also paste the table here for convenience.

As shown in the table, UFO achieves performance on par with the baseline MLLM on VQA benchmarks. We attribute this to three key factors:

1. We employed a relatively small LoRA rank (set to 8), which minimizes modifications with the pretrained MLLM parameters.
2. Our training corpus contains a substantial proportion (40%) of high-quality VQA data from LLaVA-OneVision [1], helping to preserve and reinforce general QA performance. Additionally, we observed a 0.6-point improvement on HallusionBench, which may indicate that our fine-grained fine-tuning helps reduce hallucinations.
3. UFO is highly aligned with general QA tasks, both adopting an open-ended language interface. As discussed in Lines 147-149 of the main paper, we argue that since the model can already answer where and what objects are in general QA, the mask information is already encoded in the image features. Our method can decode this information and further enhance it through training, thus essentially sharing the same capabilities as general QA tasks.

[1] LLava-Onevision: Easy visual task transfer. Arxiv 2024.

---

We hope the above response can help solve your questions. Thanks again for your thorough review and looking forward to your reply :-) !

Dear Reviewer nJnP,

Many thanks to your valuable comments and questions, which help us a lot to improve our work. We address your questions as follows.

[W1] Is it possible to integrate other visual tasks like estimating scene flow, optical flow or normals under the same framework?

Thanks for this insightful question regarding the generalizability of UFO. Our framework can naturally extend to more tasks, such as depth estimation and surface normal prediction. The core insight is to regard embedding retrieval as a general approach for extracting information from image features. While we focus on mask information extraction in the main paper, the same approach readily applies to other modalities (e.g., depth or surface normals) by introducing corresponding special tokens to interact with image tokens. We have discussed this extension to depth estimation in Appendix G, along with performance on NYUv2 Depth. For your convenience, the comparison table is also included below.

For the surface normal prediction task, we can train the model to predict three task-specific tokens: <NORMAL_X>, <NORMAL_Y>, and <NORMAL_Z>. Given the image feature $\mathbf{h}_v$ and three embeddings $\mathbf{e}_x$, $\mathbf{e}_y$, and $\mathbf{e}_z$ for each direction, the normal value (e.g., $\hat{n}_x$) is computed as follows:

$ n_x &= \sigma\left(\frac{\mathbf{e}_x^\top \mathbf{h}_v}{\sqrt{d}}\right), \hat{n}_x =2\cdot n_x-1 $

where $\sigma(\cdot)$ denotes the sigmoid function and $d$ is the feature dimension. Finally, the predicted values are normalized to obtain a unit surface normal vector:

$$\mathbf{\hat{n}} = \frac{(\hat{n}_x,\, \hat{n}_y,\, \hat{n}_z)}{\left\|(\hat{n}_x,\, \hat{n}_y,\, \hat{n}_z)\right\|}$$

We conduct the evaluation on NYUv2 Normal Benchmark, and the performance is shown in the table below. UFO can achieve comparable performance to specialized models.

A possible approach for scene flow and optical flow is to use special tokens to capture motion information from video features. However, as video training is more computationally intensive, we leave this for future work. Extending UFO to videos is highly valuable, and we will highlight this in the future work section and actively explore it!

[1] Painter: Images speak in images: A generalist painter for in-context visual learning. CVPR 2023

[2] Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Action. CVPR 2024

[3] GeoNet++: Iterative Geometric Neural Network with Edge-Aware Refinement for Joint Depth and Surface Normal Estimation. TPAMI 2020

[4] Repurposing Diffusion-Based Image Generators for Monocular Depth Estimation. CVPR 2024

[5] GeoWizard: Unleashing the Diffusion Priors for 3D Geometry Estimation from a Single Image. ECCV 2024

 

[W2] Comparison of trainable parameters when applying LoRA to train UFO.

During LoRA training, our trainable parameters include both the LoRA parameters and text embeddings. As shown in the table, UFO’s LoRA parameter count is comparable to other models and much smaller than that of SAM4MLLM. Unlike other methods, which also require training a mask decoder or even the entire LLM (e.g., HiMTok-8B, VisionLLMv2, VistaLLM), UFO achieves superior performance with fewer trainable parameters. We will add this comparison to the revision to further emphasize our advantages.

[W3] Although UFO outperforms specialist models, they have fewer parameters and may be preferred for faster inference.

Thanks for this insightful comment. The choice between UFO and specialist models depends on scenario requirements.

Specialist models are ideal for single, well-defined tasks (e.g., closed-set detection) where complex textual understanding isn't needed, as they can offer both strong performance and fast inference. In contrast, generalist models like UFO excel in complex or dynamic scenarios that require handling multiple tasks or advanced reasoning over textual instructions. As shown in Table 4 of the main paper, specialist models perform poorly on reasoning segmentation tasks compared to MLLM-based approaches.

In summary, UFO and specialist models are complementary: UFO is suited for scenarios requiring general capabilities and complex reasoning, while specialist models are preferable for straightforward, single-task problems without the need for intricate textual understanding.

[W4] Unclear writing and typos.

Thanks for these detailed observations! In Table 1, “We construct three variants by this formulation” refers to the three UFO model variants: UFO-ViT, UFO-LLaVA-1.5, and UFO-InternVL2.5-8B, each consisting of an Image Tokenizer, Text Tokenizer, and Multimodal Transformer. We have revised the manuscript to clarify this and correct all ambiguities and typos.

[Q1] Advantage of predicting box location as text compared to predicting location tokens.

Thanks for this interesting question. The main advantage of representing box coordinates with text is its simplicity and flexibility.

1. Using text representation can avoid adding thousands of new tokens to the text tokenizer, which is easier to implement and better aligned with the original MLLM.
2. If a finer coordinate granularity is needed (e.g., increasing the discrete intervals from 1,000 to 2,000), text representation allows direct reuse of prior pre-training, while location tokens would need re-initialization, making adaptation less efficient.

---

Thanks again for helping us improve the paper and hope our response can resolve your concerns! Please let us know if you have any further questions. We will be actively available until the end of rebuttal period. Looking forward to hearing from you :-) !