REN: Fast and Efficient Region Encodings from Patch-Based Image Encoders

Thank you for reviewing our work and providing constructive feedback. We are pleased to hear that you find our motivation clear and methodology well-structured. We have carefully addressed each of your comments and concerns below.

[Weakness 1] Architectural similarity to existing methods: The cross-attention module resembles DETR and MaskFormer, raising questions about novelty.

REN differs fundamentally from DETR and MaskFormer in its goal, training paradigm, design, and usage. DETR and MaskFormer are trained in a fully supervised manner for detection and segmentation, respectively, whereas REN is trained using self-supervised learning to produce efficient region-based image representations applicable across diverse tasks, including semantic segmentation, instance-level localization, image retrieval, and text-to-image retrieval.

Moreover, REN adopts a smaller and simpler cross-attention architecture, making it lighter and more efficient. Importantly, as discussed in Appendix B, REN replaces fixed learnable queries with point prompts, which has two key implications:

(1) Simplified training objective – removing the need for Hungarian matching leads to significantly faster convergence (see Figure 9); and

(2) Greater flexibility and user control – allowing region tokens to be generated for specific subsets of objects (e.g., a query object in Visual Query Localization or COCO retrieval), which is difficult to achieve with fixed learnable queries of DETR or Mask2Former.

Thus, we think the fact that REN uses cross-attention does not diminish the methodological novelty of REN.

[Weakness 2] Lacks formal proofs or guarantees for REN’s design choices (e.g., why cross-attention works for region tokenization).

While formal proofs are uncommon in deep learning, our design choices are validated through ablation studies and other experiments, which is an established and widely accepted practice.

[Weakness 3] While REN reduces SAM’s computational cost, its performance still depends on SAM-generated masks for training and certain tasks (e.g., token aggregation thresholds).

REN’s performance and the quality of its region tokens depend primarily on the feature extractor and the attention weights learned during training, not directly on SAM. SAM masks are used only to pool backbone features to generate target region tokens during training. REN does not rely on SAM masks during token aggregation or inference.

To further support that REN’s performance is independent of SAM, we highlight two key findings from our paper:

(1) Prior works (e.g., [1]) directly use SAM masks for generating region tokens, yet REN consistently outperforms them across a wide range of tasks (Section 4.2).

(2) In an experiment where SAM masks were used to directly supervise REN’s attention weights (see Appendix B), we observed reduced performance, indicating that REN benefits from learning its own attention rather than inheriting SAM’s segmentation.

[1] Region-Based Representations Revisited. CVPR 2024.

[Weakness 4] Limited exploration of alternatives: The ablation study (Section 4.3) focuses on prompting strategies but omits deeper comparisons with other attention variants (e.g., multi-head vs. single-head).

Beyond the ablation studies in Section 4.3, we explored the following alternative designs detailed in Appendix B:

(1) Supervising attention weights with SAM masks: We tested whether directly supervising attention weights using SAM masks improves performance. The result was negative—this supervision slightly degraded performance.

(2) Adapting Mask2Former for region token generation: We experimented with Mask2Former’s architecture and training objectives. While performance was comparable, this introduced unnecessary complexity, slowed training, and reduced user control.

Other design choices, such as using multi-head self-attention, are well-established, have proven to be effective in prior works, and don’t really need further validation.

Thank you for reviewing our work and providing constructive feedback. We appreciate your recognition of the paper’s clarity, the significant cost reduction, and the strong experimental results demonstrating the method’s effectiveness. We have carefully addressed each of your comments and concerns below.

[Weakness 1] The essence of REN's training is to establish the correlation between points, but the association between points and regions still seems to rely on other models for assistance, which restricts its application scope and makes it impossible to truly achieve real-time performance.

As shown in Table 1, REN processes images at more than 30 FPS, demonstrating that it is capable of real-time performance. Importantly, REN only uses external masks from SAM during training. At inference, REN operates independently of any segmentation model—region tokens are generated directly from the backbone features and point prompts, making the method lightweight and efficient.

[Weakness 2] As described in lines 108-111 of Table 1, the RPN utilizes the prior information from SAM 2. I am curious whether this makes the training constitute an implicit distillation of SAM 2, which is why the RPN has performance advantages? Please replace the priors of other models or directly discuss this point.

There is a typo in Section 3: we use SAM, not SAM 2, to generate region masks for training. This choice was made to ensure a fair comparison with baselines [1, 2], which also use SAM-generated masks. We appreciate you bringing this to our attention and we will correct it in the revision.

[1] Region-Based Representations Revisited. CVPR 2024.

[2] RELOCATE: A Simple Training-Free Baseline for Visual Query Localization Using Region-Based Representations. CVPR 2025.

[Weakness 3.1] Compared with DETR and Mask2Former, the RPN method is slightly less innovative, as it merely replaces learnable queries with point prompts.

REN is fundamentally different from DETR and Mask2Former in its goal, training paradigm, design, and usage. DETR and Mask2Former are trained in a fully supervised manner for detection and segmentation, respectively. In contrast, REN is trained using self-supervised learning to produce efficient region-based image representations applicable across diverse tasks, including semantic segmentation, instance-level localization, image retrieval, and text-to-image retrieval. REN also employs a smaller and simpler cross-attention architecture, making it more lightweight and efficient.

Furthermore, as discussed in Appendix B, using point prompts instead of learnable queries has significant implications:

(1) Simplified training objective – removing the need for Hungarian matching leads to faster convergence (see Figure 9); and

(2) Greater flexibility and user control – allowing region tokens to be generated for specific subsets of objects (e.g., a query object in Visual Query Localization or COCO retrieval), which is difficult to achieve with fixed learnable queries of DETR or Mask2Former.

[Weakness 3.2] In addition, I am curious whether the region representations obtained by RPN are superior to those obtained by methods such as DETR and Mask2Former?

Unlike REN, DETR and Mask2Former are not designed to produce general-purpose region-based representations, but are specialized for detection and segmentation. As a result, REN’s region representations are more broadly applicable across diverse tasks (ref. Section 4.2). To further examine this, as discussed in Appendix B, we adapted the Mask2Former architecture for region representation learning. While it achieved performance comparable to REN, it required significantly more complex optimization, leading to slower training (see Figure 9). Moreover, its reliance on fixed learnable queries limits user control, making it challenging to generate region representations for specific objects within an image—a capability that REN’s point prompting naturally supports.

[Weakness 4] Can the proposed method be directly used for fine-tuning like [1], and bring performance advantages to fine-grained tasks?
[1] UMG-CLIP: A Unified Multi-Granularity Vision Generalist for Open-World Understanding. ECCV 2024.

We think it’s possible; great suggestion for future work.

Dear Reviewer,

We just wanted to kindly follow up on our earlier message. With the discussion period ending soon, we wanted to check if you had any further questions or concerns regarding our rebuttal. We’d be happy to clarify anything that remains unclear.

Authors of REN

Thank you for reviewing our work and providing constructive feedback. We appreciate your recognition of REN’s creative design, efficiency, compact region tokens, and strong performance validated by comprehensive experiments. We have carefully addressed each of your comments and concerns below.

[Weakness 1 & Question 2] Fig. 1 can be redrawn to better epitomize the implementation details. The mask generated by SAM2 and the definition process of positive and negative samples should be added.

Thank you for the suggestion. We will update Figure 1 to include the missing training components—specifically, the SAM-generated masks, the target region tokens used for feature similarity loss, and the positive/negative samples used for contrastive learning.

[Weakness 2] This paper discusses images with few objects in Intro and argues it as a reason which causes the inefficient use of current patch tokens generation. Although REN shows its efficiency in such images, it lacks analysis about images with dense objects.

Our argument in the intro is that the number of tokens needed to represent an image should depend on its complexity, which is not the case for current patch-based methods. For example, on Pascal VOC, we can represent images using fewer than 30 tokens on average while achieving performance comparable to a patch-based counterpart that uses 1,369 tokens per image. A similar analysis on ADE20K shows that we achieve comparable performance using 41 tokens per image instead of 1,369 (Figure 7). This demonstrates that REN allocates more tokens as scene complexity increases (30 → 41 on average), enabling efficient handling of both simple and dense scenes.

Furthermore, REN effectively handles highly complex and dense images as well. This is evident from our results on the challenging Ego4D VQ2D task, which involves localizing small, overlapping, and occluded objects in cluttered scenes (see Figure 4). REN significantly outperforms all prior methods, including those specifically designed for this task.

[Question 1] In Section Intro, “existing SAM-based methods for region token ...., which can remove fine-grained details and .... Current segmentation methods also .....”, how can you make sure that it’s the simple linear aggregation operation that causes the over-representation and no-representation phenomenon?

Simple linear aggregation of features treats all pixels equally, which can dilute important details—over-representing irrelevant regions and under-representing key discriminative areas. Additionally, restricting feature aggregation strictly to the object region can discard valuable contextual information. This effect is reflected in our results in Tables 7 and 10, summarized below:

Furthermore, REN consistently outperforms SAM-based methods—which rely on simple linear aggregation—across a wide range of tasks (Section 4.2), reinforcing that learned discriminative aggregation leads to superior region representations.

[Question 3] In Section 4.3, “all ablations are conducted using REN trained with DINOv2 as the image encoder.”, since DINOv2 with registers reports its segmentation improvement, have you ever tried to use DINOv2 with registers as your encoder?

We conduct additional ablations using DINOv2 with registers as the image encoder, focusing on semantic segmentation on the Pascal VOC validation set. The results are consistent with the ablations reported in the paper:

Training loss ablation: A combination of contrastive and feature similarity loss performs the best.

Effect of prompting strategy and token aggregation: SLIC-based prompting provides slightly better performance, and token aggregation maintains accuracy while reducing the number of tokens required to represent each image.

As expected, using DINOv2 with registers provides a marginal performance improvement:

These results show that REN’s performance scales with backbone improvements for the given task while maintaining its core performance trends and ablation conclusions.

[Question 4] Some descriptions in this paper are confusing. In Abs and Intro, this paper uses “region-based image representations”, “region representations”, “region tokens” and “region-based representations”. Are there any differences among these words? Or some of the words share the same meaning?

All these phrases mean the same thing. We will clarify this better in the revision.

Thank you for reviewing our work and providing constructive feedback. We appreciate your recognition of REN’s efficiency, broad encoder compatibility, and strong performance across multiple benchmarks. We have carefully addressed each of your comments and concerns below.

[Weakness 1] While REN is described as "segmentation-free," the training process relies on pseudo-labels generated by SAM, implying an indirect dependency on segmentation and possible interpretation as knowledge distillation from SAM.

When we describe REN as “segmentation-free,” we refer to its inference phase: region tokens are generated directly from the trained REN model without performing any explicit segmentation step, unlike prior works. While pseudo-labels from SAM are used only during training to guide learning, this does not impose a segmentation requirement at inference. For instance, all downstream tasks in Section 4.2 operate without any segmentation step, resulting in significantly lower computational time and memory usage.

[Weakness 2] Although the authors emphasize the importance of preserving alignment with the original encoder via feature similarity loss, the ablation study is limited and lacks quantitative analysis of its impact on downstream tasks.

The ablation study presented in Table 7 (copied below for quick reference) already demonstrates that using both feature similarity and contrastive loss yields the best overall performance. To further quantify the benefit of feature alignment, we additionally report results on the COCO retrieval task: the mRP is 0.56 when only contrastive loss is used, and it is 0.65 when feature similarity loss is also used (higher is better). This now constitutes an ablation across all downstream tasks discussed in the paper, except for Visual Haystacks (because there REN is used purely as a segmentation model to guide feature pooling from SigLIP 2 features).

[Weakness 3] The self-supervised region token learning in REN, based on contrastive learning for object-level consistency, can be viewed as an incremental extension of prior works such as TokenCut and LOST, rather than a fundamentally novel paradigm.

We respectfully disagree. TokenCut and LOST are training-free methods designed specifically for object localization in images. In contrast, REN is a trained model that learns generalizable image representations applicable to a wide range of tasks, including semantic segmentation, instance-level localization, image-based retrieval, and text-based retrieval. The only commonality is that both approaches leverage object-level cues present in patch features from self-supervised Transformer models (as noted in lines 81–83), but REN’s goal, training paradigm, and capabilities differ fundamentally from TokenCut and LOST.

[Weakness 4.1] As acknowledged by the authors, there exists inherent ambiguity regarding whether a single point prompt represents the entire object or only a part, requiring heuristic adjustment of the aggregation threshold (µ) based on the task.

We have acknowledged this as a limitation. However, as shown in Figure 7, REN’s performance is robust across a wide range of aggregation thresholds (µ), indicating that the model is not highly sensitive to this parameter in practice.

[Weakness 4.2] The point prompting strategy may lead to reduced accuracy in complex scenes containing overlapping objects, where precise region separation is more challenging.

We think the point prompting strategy is more adaptive and better suited for complex scenes, as the number of point prompts can be increased to capture finer object details in dense regions. This advantage is evident in our results on the challenging Ego4D VQ2D task, which involves localizing small, overlapping, and occluded objects in highly cluttered scenes (see Figure 4). REN significantly outperforms prior methods, including those specifically designed for this task, highlighting its effectiveness in handling overlapping objects and complex layouts. Moreover, our segmentation results on ADE20K, where REN surpasses patch-based baselines, further support this conclusion.

[Weakness 6] Minor Typo: Line 303, "limiting it’s" → "limiting its".

Thanks for pointing it out; we will fix it in the revision.

Dear Reviewer,

As the discussion period closes in less than two days, we’d be very grateful if you could confirm whether our rebuttal addressed your concerns. Your response would help ensure a fair and informed decision. If any issues remain, we’d be happy to clarify them promptly.

Authors of REN