R-MAE: Regions Meet Masked Autoencoders

1. Self-supervised contrastive learning needs to introduce the concept of region to focus local information due to its a priori assumption of image semantic consistency, but MAE does not have this problem. Moreover, I agree that the reconstruction of raw pixel values lacks a higher level of semantic information for image understanding compared to word reconstruction in NLP. However, I do not agree that the introduction of binary regions adds high-level semantics. Therefore, I argue this paper is the same as previous self-supervised object detection learning. The effect comes from further learning of the local region, so it is effective in tasks such as detection and segmentation.

• Thanks for the feedback. While we indeed wanted to pursue the visual analogue of words as our goal, we also understand the concern that regions do not automatically translate to high-level semantic information. On the other hand, regions do offer resemblances to words that encode discrete information about which set of pixels should be in the same “group”, and we believe it’s a better way to phrase it. We have updated our introduction to reflect this, and added another limitation to reduce confusion. Please see page 1 and page 9 of the latest draft.

2. Like question 1, the paper lacks ImageNet classification experiments.

• We have included ImageNet classification results as part of our supplementary material, we make a copy below for convenience:

• In the fine-tuning setting, we find MAE and R-MAE to be at-par. For linear probing, we do find MAE to be more effective.
• Nevertheless, we are no longer claiming regions introduce semantics into MAE pre-training, and believe our exploration and discovery are still valuable.

3. The calculation amount comparison is unfair. The paper only calculates the calculation amount of the architecture, but the results of the FH algorithm and SAM region generation are not included.

• Thanks for pointing this out. We have added a footnote on page 7 to address this concern. Indeed, R-MAE needs extra overhead to compute regions. Yet FH [R1] is designed to be highly efficient and has a complexity of O(n log n) where n is the number of pixels. Moreover, regions are only computed once per image and can be pre-computed with CPUs. In practice, we do not observe any overhead when loading it from the disk.

• As for SAM, it’s indeed expensive (added a note on page 9), but its main purpose in our paper is to serve as an oracle that provides upper bounds. Following FH, we also pre-computed SAM regions and loaded them during pre-training. We look forward to more studies on making it more efficient.

[R1] Felzenszwalb, Pedro F., and Daniel P. Huttenlocher. "Efficient graph-based image segmentation." International journal of computer vision 59 (2004): 167-181.

4. From Table 3, more data show that the gain in segmentation results relative to MAE is weaker. Is there no difference between the results of MAE and R-MAE on a larger data set?

• For mIoU, semantic segmentation on ADE20K is in general less reliable as a trend indicator. For example, see Fig. 4a and Fig. 4b as we reported both AP$_b$ and mIoU. While the trend in AP$_b$ is smooth, in mIoU there are outliers even though we average over multiple seeds for each point on the plot. We hypothesize it’s due to the size of the dataset.
• For AP$_m$, we also reported pre-training on ImageNet which is 10x larger, and still observed healthy gains over MAE, copied below. The gain on AP$_m$ is increased from COCO++ to ImageNet from 0.2 to 0.3. Here, R-MAE are pre-trained with FH regions:

Here, COCO means COCO train set and COCO++ means COCO train + unlabelled set

• In general, we believe the trend is also dependent on other factors (e.g., types of data, types of masks used given the strong results with SAM regions), and may not be simply concluded from results of a single experiment.

Since this is a multi-folded question, we address them with bullet points:

1. About the importance of regions:

• While regions can be generally interpreted as groups of pixels given an image, we agree a more rigorous definition would be helpful. We also fully agree that the most suitable regions can vary from task to task, but we also want to point out the value of finding the right regions for important tasks. For example, physiological theories [R1] suggest that human perception will group similar elements and parts together to parse complex scenes and objects. It would be of tremendous value to discover such regions, not only for the scientific understanding of intelligence, but also for practical applications that interact with humans.
• Yes this is an intriguing empirical result. One hypothesis is that SAM covers a more diverse set of regions (e.g., not just the entire area of a human, but also parts such as faces and clothes) beyond the ~200 classes defined for panoptic segmentation; the other hypothesis is that SAM regions are more consistent and high in recall, whereas human annotations can be subject to disagreement and carelessness. As written in our last paragraph, we are also curious to test these hypotheses and simplify/accelerate our pipeline as future work.
• Thanks for pointing this out. We provided the comparisons on ViT-L as part of our supplementary material (see Table C), which has even less relative extra computation from the RAE branch. However, we do agree that the tone in the introduction needs a revisit. We have made updates to the introduction and added another limitation to specifically address this concern. Please check them out.

[R1] Koffka, Kurt. Principles of Gestalt psychology. Vol. 44. Routledge, 2013.

2. About the architecture:

• Sorry for the confusion, but no, the region encoder only operates on binary region maps and does not share weights with the pixel encoder.
• Both RAE and R-MAE have the pixel encoder. In the case of RAE, the pixel encoder gets pre-trained from reconstructing regions as it feeds into the region decoder.
• The pixel encoder gets transferred to downstream tasks. It has seen RGB images before and can in fact be directly used as an interactive segmenter.
• All tasks we have experimented with are color-sensitive ones, including COCO/LVIS object detection and segmentation, and ADE20K semantic segmentation. However, we are happy to conduct more classification experiments (e.g., on Flowers) if the reviewer feels the need.
• Thanks for the suggestion on joint-branch training. Indeed it’s a valid idea and if we understand correctly, has at least been explored in SemMAE. We compare with SemMAE in Table 5, in which we show advantage in both efficiency and effectiveness.
• Maintaining a separate branch for the region encoder is especially helpful when there are a large number of regions per-image. The size of the region encoder can then be decoupled to the pixel encoder, and maintain permutation equivariance as we did in our "length-variant".
• We also empirically find the asymmetric design that feeds pixel encoder for regions but not vice versa works best. Using region encoders to predict pixels is not as effective, as shown in Tab. 1e.

3. R-MAE has been surpassed by earlier MAE-based framework targeting at detection and segmentation with local awareness (e.g., 800-epoch MixedAE [2] outperforms 1600-epoch R-MAE on ADE20K).

• Thanks for the pointer. We have updated our draft and included MixedAE [2] in our comparison (Tab. 5). Please check out page 8.
• We find MixedAE works well for semantic segmentation and has acknowledged it in the main text. However, we do also find our R-MAE results on COCO to be more competitive (according to Tab. 1 of [2]) and with less per-iteration compute during pre-training.

4. It would be better to also report quantitative comparison for interactive segmentation for better understanding in Fig. 6.

• Thanks for the suggestion. We computed the mIoU between the predicted regions from R-MAE and ground-truth regions on COCO for a subset of its validation images. The results are shown below:

• With mask ratio 90%, RAE can already reach an mIoU of 76%. Note that this is highly competitive as a promptable segmenter according to the SAM paper (Fig. 9c and Fig. 9d), where even the initial oracle is below 75%.
• As expected, more visibility leads to better results. This is also consistent with our visualizations.

1. Could the authors provide a more extensive quantitative comparison with other state-of-the-art self-supervised learning methods in computer vision? This would help readers understand how R-MAE performs in relation to existing techniques

• In Table 5, we have compared R-MAE to several state-of-the-art reconstructive pre-training methods when pre-trained on the standard ImageNet dataset. To make it more comprehensive, we added more results from:

  • (1) a latest work published at CVPR 2023 (MixedAE);
  • (2) supervised pre-training;
  • (3) contrastive pre-training (and more) to the draft.

• The results of (2) and (3) can offer a more complete picture of other self-supervised learning methods on this task. Please check page 8 of the updated draft for the results and the accompanied descriptions.

2. Is the performance of R-MAE sensitive to the quality of region maps, and are there strategies to mitigate this sensitivity?

• In Tab. 1d, we show results with regard to different sources of regions (to better capture the sensitivity, we conducted this analysis on RAE without the MAE branch). Several remarks:

  • High-quality region maps indeed help a lot! With pre-computed SAM regions (as an oracle), RAE alone can achieve better results than MAE.
  • RAE also works gracefully when the region quality degenerates, e.g., with FH.
  • One interesting observation is that RAE favors regions that densely cover the whole image. For example, RAE does not work well for ground-truth COCO instance annotations (AP$_b$=44.7, AP$_m$=39.5, mIoU=41.6). However, dense coverage requires our model to be highly efficient with a large number of regions per-image (e.g., 54.5 on average for SAM regions). Therefore, our length-variant can be viewed as an effective mitigation strategy.

• In general, R-MAE works with various types of regions from a simple graph segmentation algorithm like FH to deep learning models like SAM.