Perceive Anything: Recognize, Explain, Caption, and Segment Anything in Images and Videos

Response to W1: Performance gains of PAM may not be attributed to its architectural design，but rather from its utilization of extensive labeled data

Data is one of the key contributions of our work. We identified shortcomings in open-source data annotations and performed a comprehensive refinement. We acknowledge that this refined data provided a significant boost to PAM’s performance.

However, for the classification and OCR tasks in Table 1, we used the original open-source data. On these benchmarks, our PAM-3B model achieves SOTA results on LVIS (semantic IoU), PACO, and Total-Text. Notably, all competing models we compared against are 7B or larger in scale. This demonstrates the effectiveness of the PAM architecture itself, independent of the data enhancements.

Response to W2: Comparison with Existing general VLMs.

Thank you for pointing this out. To demonstrate the necessity of region-level VLMs, we compared them against general VLMs on the DLC-Bench[1], which evaluates accuracy on detailed localized captioning. The experimental results are as follows:

As the results indicate, Region-level VLMs significantly outperform general VLMs on fine-grained detailed captioning tasks. This validates the necessity of research into region-level VLMs.

[1] Describe Anything: Detailed Localized Image and Video Captioning

Response to W3: Explanation of Efficiency Gains

Thank you for your insightful comments. To better demonstrate the efficiency of our model, we trained a PAM-7B to compare it directly with other 7B models. This approach allows us to rule out the influence of model size and focus on architectural advantages.

The experimental results show that PAM-7B still outperforms others in terms of efficiency. This superiority can be attributed to three key factors:

1. SAM 2, which serves as PAM’s vision encoder, is more lightweight compared to the encoders used in other VLMs.

2. Fewer Visual Tokens: The number of visual tokens processed from SAM 2 and fed into the LLM is significantly lower (1024 vs. more than 2048) than in other models.

3. PAM utilizes the latent features from SAM 2, so it does not need to wait for the entire SAM 2 inference process to complete. Once the latent features are obtained, mask decoding and semantic decoding can proceed in parallel.

Dear Reviewer R2WL,

Many thanks for your time in reviewing our paper and your constructive comments. We have submitted the point-to-point responses on attribution of performance gains, comparison with general VLMs, and explanation of efficiency gains. We would greatly appreciate it if you could let us know whether your concerns have been addressed. We are also happy to provide further clarifications if needed.

Best regards,

Authors of paper #3605 Perceive Anything: Recognize, Explain, Caption, and Segment Anything in Images and Videos

Response to W1: Lacks support for more general vision-language tasks like open-ended (VQA)

Although PAM specializes in providing accurate and detailed region-level descriptions, its core architecture inherently supports other understanding tasks, such as visual question answering (VQA), without any modification.

To demonstrate this capability, we trained our model on several open-source, region-level VQA datasets, including MDVP-Instruct, VCR, Visual7W, and Osprey-724K. The training data was formatted as question-answer pairs targeting a specific area, for example: “What kind of clothes is the man wearing in this <region>?”

The experimental results are as follows:

Response to W2: Concerns about the generalizability and scalability

Thank you for pointing this out. Our initial objective was to train a region-level VLM that is fast, compact, and high-performing, making it easier to deploy in practical, real-world scenarios. At the same time, we recognize that scalability is crucial. Therefore, we dedicated additional time and resources to train PAM-7B. The experimental results are as follows. As can be observed, PAM-7B exhibits outstanding performance, which demonstrates that our method possesses robust generalizability and scalability. Looking ahead, if sufficient computational resources are available, we will involve testing a 32B model to further evaluate its scalability.

Dear Reviewer y2LG,

Many thanks for your time in reviewing our paper and your constructive comments. We have submitted the point-to-point responses on the ability to more general vision-language tasks, and the generalizability and scalability. We would greatly appreciate it if you could let us know whether your concerns have been addressed. We are also happy to provide further clarifications if needed.

Best regards,

Authors of paper #3605 Perceive Anything: Recognize, Explain, Caption, and Segment Anything in Images and Videos

The authors have solved most of my questions. Therefore, I choose to keep my score.

Response to W1 and Q1

Thanks for your comments. However, we think this concern is based on a misunderstanding of our research. Here are the reasons:

1. Does the model lost its instruction following ability after such training?

Our goal was to align the text-only LLM (Qwen2.5) with SAM 2 to create a VLM that excels at powerful region-level visual caption. After training, PAM is endowed with new visual instruction-following abilities, including open-set classification, functional explanation, brief and detailed captioning, and streaming video captioning. Therefore, the LLM’s original ability to follow text instructions is not a key factor in PAM.

2. The LLM could lost its instruction following ability and cannot give perception result for text beased prompts.

Using text-based prompts to identify objects is a feature of general VLMs; the Qwen2.5 LLM, however, never had this capability, as it is unable to process multimodal data. Moreover, we designed PAM to use visual prompts (points, boxes, masks) from SAM 2 because they offer higher precision in complex scenes where describing an object with text can be difficult and ambiguous.

3. It will lost the LLM's ability as foundation model, which highly limits the application of this framework compared to methods like DAM.

We think that by integrating two powerful foundation models (SAM 2 and Qwen2.5), we can create a new and robust region-level vision-language foundation model, excelling in fine-grained visual understanding. Our results show this specialized model achieves superior performance on multiple tasks compared to other methods.

For DAM, it is also specialized, focusing primarily on captioning tasks while not supporting others like VQA. Furthermore, DAM is less flexible, as it only accepts mask-based visual inputs. In our experiments, PAM-3B outperforms DAM-3B on multiple benchmarks, demonstrating its effectiveness.

Response to W2 and Q2: Why is SAM2 frozen?

Here are the reasons:

1. The design of PAM, has two main goals: (1) to generate precise masks based on visual prompts, and (2) to provide rich semantic information for those masks. If it was unfrozen and retrained, the SAM2’s existing segmentation capability could be degraded. While it’s possible to train both SAM2’s segmentation and the LLM’s visual understanding simultaneously, this approach would not only require significantly more computational resources but also introduce a more complex multi-task learning challenge, making the model more difficult to optimize. Therefore, freezing SAM2 is a far more efficient strategy for the overall training process.

2. SAM2 inherently contains rich pixel-level information. A key part of our work was to figure out how to best select latent features from SAM2 to efficiently train a region-level VLM. In our ablation studies, we investigated how choosing different latent features impacts model performance, and we analyze the potential reasons in the Methods section. Our experiments demonstrate that it is possible to achieve a high-performing PAM model without retraining SAM2.

3. To further validate this, we ran an additional experiment. We trained PAM-1.5B from stage 1.5 checkpoint on 0.5M region-level classification data under two conditions: once with SAM2's parameters frozen, and once with them unfrozen. The results showed that unfreezing SAM2 offered no significant performance boost.

Response to W3 and Q3: Can this framework generalize to other understanding tasks?

Although PAM was designed to specialize in accurate and detailed region-level descriptions, its architecture inherently supports other understanding tasks (e.g., visual question answering) without requiring any modifications.

To demonstrate this, we collected and trained our model on several open-source, region-level VQA datasets, such as MDVP-Instruct, VCR, Visual7W, and Osprey-724K. The training data were constructed in a format like, "What kind of clothes does the man wear in this <region>”

The experimental results are as follows:

Dear Reviewer H65M,

Many thanks for your time in reviewing our paper and your constructive comments. We have submitted the point-to-point responses on instruction-following ability, why freezing SAM2, and generalization. We would greatly appreciate it if you could let us know whether your concerns have been addressed. We are also happy to provide further clarifications if needed.

Best regards,

Authors of paper #3605 Perceive Anything: Recognize, Explain, Caption, and Segment Anything in Images and Videos