Multi-Scale Fusion for Object Representation

Thank you for your feedback.

Weakness 1

According to Sec. 4.1, the improvements from MSF are limited, just 1-2 points in many metrics.

We would like to clarify possible misunderstandings.

Techniques OGDR and MSF in Tab. 1 are added to SLATE separately, i.e., the table contains results for SLATE+OGDR and SLATE+MSF, not SLATE+OGDR+MSF together. Therefore, MSF improves SLATE 5-7 points in both ARI and ARIfg, that is, 24.18->30.95@ARI and 24.54->30.47@ARIfg.

Besides, the methods we cover are all strong baselines, and any improvement can be challenging. We exclude earlier methods like IODINE, MONet, SA and SAVi due to their lower accuracy. In OCL literature, ARI and ARIfg (2nd and 3rd columns in those tables) are widely adopted metrics, revealing more significant differences among methods.

Weakness 2

The overall presentation is not satisfactory and difficult to understand for readers not familiar with OCL.

We have improved the presentation in our paper's new version.

And we would be happy to polish it further if you could provide specific examples or clarify which parts are unclear.

Question 1

Can the proposed method be applied to state-of-the instance segmentation models, like MaskDINO, to improve their results (~50 mask AP) on MS-COCO for instance segmentation.

Our MSF only modifies VAE's codebook quantization part, making it applicable to any model using codebook quantization. Unfortunately, MaskDINO does not have such a module. This requires further investigation in the future.

Dear Reviewer 8Piy, following up on our response to your valuable feedback.

Your comments have been greatly appreciated and have helped refine the work.

If you had a chance to review the rebuttal, we would be grateful for any additional thoughts or clarifications, especially if there are unresolved concerns that we can address to improve the paper further.

Thank you again for your time and insights, looking forward to hearing from you soon.

Thank you for your feedback.

Weakness 1

Although I am not familiar with object-centric learning, I believe that constructing multi-scale VAE features has already been applied in many generative tasks[1, 2]. Can the author tell me about the difference between MSF and other methods?

Please note that we have updated the related contents into Sect. "A.1 Extended Related Work" of our paper’s new version.

Short answer

• VAR [1] vs MSF: VAR auto-regresses from smaller scales of VAE representation to the larger, but with no information fusion among scales. In contrast, our MSF realizes fusion among different scales on VAE for the first time.

• SPAE [2] vs MSF: SPAE relies on multi-modal foundation model CLIP, while our MSF does not. SPAE element-wisely averages multiple scales into one, simply mixing different scales together, i.e., no fusion among scales. Our MSF augments any scaled representation with all other scales' high-quality representations, i.e., fusion among scales.

Long answer

[MSF] interpolates the input and produces multiple scales of VAE representation, which augment one another. Every scale integrates high-quality representations from the other scales, yielding better visual explanation (Fig. 3). Specifically,

(1) The input image $T$ is spatially interpolated into a pyramid {$T_n$}, from high to low resolutions, and after VAE encoding, we have multiple scales of VAE intermediate representation {$Z_n$};

(2) We project it into two copies, one for scale-invariant quantization using a shared codebook, yielding {$X_n^{si}$}, and the other for scale-variant quantization using different codebooks, yielding {$X_n^{sv}$};

(3) We conduct our unique and novel inter-scale fusion and intra-scale fusion upon {$X_n^{si}$} and {$X_n^{sv}$}, yielding the augmented {$X_n$}, where whichever scale is augmented by the other scales with their high-quality representation, i.e., fusion among the scales;

(4) The final {$X_n$} is used to guide OCL training.

[VAR] interpolates the single VAE intermediate representation and produces multiple scales of VAE representations, where there is no augmentation in between. Specifically,

(1) The input image $im$ is VAE encoded once, yielding one single intermediate representation $f$;

(2) The authors spatially interpolate $f$ into multiple scales and residually quantize them with a shared codebook, yielding a sequence of discrete representations $R=[r_1, r_2,... r_K]$, from low to high resolutions;

(3) There is no fusion among the scales;

(4) Finally the authors conduct auto-regression upon $R$ for image generation, i.e., $p(r_1, r_2, ... r_k) = \Pi p(r_k | r_1, r_2,... r_{k-1})$.

[SPAE] interpolates one VAE intermediate representation and yields multiple scales of VAE representations, but employs naive element-wise-average to integrate those scales into one. Specifically,

(1) The input $I$ is VAE encoded once, yielding one single intermediate representation $Z$;

(2) The authors conduct spatial interpolation on it and yield {$Z_l$}, and use pretrained multi-modal foundation model CLIP to residually quantize them under some prompt, producing a sequence of lexical tokens {$\hat{Z}_l$}, from low to high resolutions;

(3) The authors streaming average the first $l$ scales into one representation $\hat{Z}\_l$ – They have no fusion among the scales; Instead, they just mix all scales together.

(4) Finally the mixed representation is VAE decoded to produce the reconstruction or image generation.

Summary

Whether with VAE or not, it is not difficult to build multiple scales of representations; Outside of VAE, there are also many techniques to fuse multiple scales. But, our MSF is the first to both build multiple scales upon VAE and realize fusion among multiple scales, using unique and innovative designs based on codebook matching.

Reference

[1] Tian et al. Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction. NeurIPS 2024.

[2] Yu et al. SPAE: Semantic Pyramid AutoEncoder for Multimodal Generation with Frozen LLMs. NeurIPS 2023.

Thank you for your feedback.

Weakness 1

Results on real world datasets especially COCO in Table 1 are incremental. Not really clear how much advantage MSF adds.

We would like to clarify possible misunderstandings.

Techniques OGDR and MSF in Tab. 1 are added to SLATE separately, i.e., the table contains results for SLATE+OGDR and SLATE+MSF, not SLATE+OGDR+MSF together. Therefore, MSF improves SLATE 5-7 points in both ARI and ARIfg, that is, 24.18->30.95@ARI and 24.54->30.47@ARIfg.

Besides, the methods we cover are all strong baselines, and any improvement can be challenging. We exclude earlier methods like IODINE, MONet, SA and SAVi due to their lower accuracy. In OCL literature, ARI and ARIfg (2nd and 3rd columns in those tables) are widely adopted metrics, revealing more significant differences among methods.

Weakness 2

Analysis of varying object sizes can show results on scale understanding better than overall IoU improvement.

We follow COCO's small/medium/large size splits to evaluate our MSF's performance, as shown in the table below. MSF shows more improvement on small and medium objects. This demonstrates that the fusion among multiple scales effectively improves different-sized object representations.

Table 1. How MSF performs on different-sized objects. Dataset is COCO instance segmentation.

Please note that the related contents have been updated into Sect. "A2. Extended Experiments" in our paper’s new version.

Weakness 3

Is there any intuition why the value of n is 3? Can we do a similar experiment on OpenImages subset [1] and see if this holds true across datasets?

We use input resolutions 128 and 256 to evaluate on OpenImages subset. Under resolution 128, we use $n$=3 and 4; under resolution 256, we use $n$=3, 4 and 5. Results are shown below. We use ARI+ARIfg as the metrics because ARI mostly reflect how well the background is segmented and ARIfg only measures foreground objects. According to the results, under resolution 128, n=3 is the best choice; under resolution 256, n=4 is the best choice. $n$ value is stable across datasets.

Table 2. Effects of input resolutions and #scales $n$. Model is SLATE+MSF; dataset is OpenImages subset.

Please note that the related contents have been updated into "A.2 Extended Experiments" of our paper’s new version.

Weakness 4

Discussion on the area of unsupervised semantic segmentation and object detection using SSL methods should be added. Papers like [1,2,3] should be discussed.

Firstly, relationships among SSL segmentation (SSLseg), OCL, as well as World Models (WMs) are as follows. (1) SSLseg focuses on extracting segmentation masks; (2) OCL represents each object as a feature vector, with segmentation masks as byproducts, kind of overlapping with SSLseg; (3) WMs, upon OCL, address downstream tasks like visual reasoning, planning and decision-making.

Briefly, OCL and SSLseg are designed for different purposes. OCL can be used directly to support WMs to work on visual tasks like reasoning, planning and decision-making; In contrast, SSLseg needs some intermediate step like OCL to support WMs on those advanced vision tasks.

Please read Fig. 4 in our paper's new version, which provides a nice picture to show the relationships intuitively.

Since SSLseg and OCL are designed for different purposes, direct comparisons are not commonly discussed in the OCL literature – We follow the evaluation protocols established by [4, 5, 6].

Please note that we have updated the related contents into our paper’s new version.

Reference

[1] Ge et al. Hyperbolic Contrastive Learning for Visual Representations beyond Objects. CVPR2023.

[2] Wang et al. VideoCutLER: Surprisingly Simple Unsupervised Video Instance Segmentation. CVPR 2024.

[3] Rambhatla. MOST: Multiple Object Localization with Self-Supervised Transformers for Object Discovery. ICCV 2023.

[4] Locatello et al. Object-Centric Learning with Slot Attention. NeurIPS 2020.

[5] Singh et al. Illiterate DALL-E Learns to Compose. ICLR 2022.

[6] Kipf et al. Conditional Object-Centric Learning from Video. ICLR 2022.

Thank you for your positive feedback.

Weakness 1.1

The ideas of multi-scale are not novel (e.g., FPN), but implementation and application to OCL are.

We agree that multi-scale is a key challenge in computer vision.

Existing multi-scale methods (i) Either employ image pyramids without information fusion among pyramid levels, where inter-scale information is wasted; (ii) Or rely on feature pyramids (FPN and numerous variants), where features of different layers are fused by channel-concat or element-wise-sum, mixing both low- and high-quality representations together.

Although our main focus is on the OCL setting, our MSF is the first to enable multi-scale fusion on VAE representations. By leveraging codebook matching, we selectively fuse high-quality information among scales, rather than mixing them together. Our MSF is superior to channel-concat or element-wise-sum, as shown below.

Table 1. Effects of different fusion techniques among multiple scales. Model is SLATE; dataset is ClevrTex.

Please note that the related contents have been updated into Sect. "A.1 Extended Related Work" of our paper's new version.

Weakness 1.2

The implementation neglects potentially more impactfully redesigns.

We are open to including other impactful OCL designs if specific examples are provided. Regarding earlier works like IODINE, MONet, SA, and SAVi, we omitted them due to their lower performance.

Weakness 2

No results for downstream tasks.

Downstream tasks of OCL (+MSF) include (1) scene understanding, e.g., VisualGenome; (2) visual reasoning, e.g., Clevrer VQA; (3) visual planning, e.g., PHYRE and Physion; and (4) visual decision-making for reinforcement learning agents, e.g., video game playing and robotic manipulations.

OCL literature usually does not cover these tasks. However, we provide results for visual planning in PHYRE with a SlotFormer [2] World Model (WM) below.

Table 2. Visual planning on PHYRE [1]. OCL extracts slots; WM infers upon slots. AUCCESS [2] is the metric, the larger the better.

Our MSF also has potential in visual generation. This is because we only modify VAE, the key module of visual generation like StableDiffusion for images and Sora for videos. However, this is a totally different research field, beyond our scope.

Weakness 3

Minor: The notation and its interaction with Figure 3 is difficult to follow. The differences between scale-variant, scale-invariant, and representation before inter-scale fusion are quite slight.

We have improved Fig. 3 presentation, especially the figure caption, in our paper’s new version. Please check the new pdf file for details.

And we are happy to improve further. Could you kindly provide specific examples?

Weakness 4

Minor: Algorithm 1 would fit in the main paper, and help add clarity in the flow.

Thank you for your kind advice. We have updated our paper’s new version accordingly.

Question 1

What downstream tasks could this method help with? It doesn't seem to be a SAM alternative, so what is the potential practical impact?

Segmentation models like SAM only extract masks, but cannot represent objects as corresponding feature vectors, i.e., slots. In contrast, OCL can do both.

With slots from OCL, an object-centric world model can be built for downstream tasks like visual understanding, reasoning, planning and decision-making [2]. Our MSF improves OCL, thus helps with all of these tasks.

Question 2

What would change were this applied to higher-resolution images? MSF would seem to have some promise for small objects.

We follow COCO's Small/Medium/Large size splits to evaluate our MSF's performance. Results of resolution 128x128 and 256x256 are shown below. Our MSF does show better performance on small objects; And switching to higher-resolution does not change the conclusion.

Table 3. How MSF performs on different-sized objects. Dataset is COCO instance segmentation.

Please note that the related contents are updated into Sect. "A.2 Extended Experiments" of our paper’s new version.

Reference

[1] Bakhtin et al. PHYRE: A New Benchmark for Physical Reasoning. arXiv:1908.05656.

[2] Wu et al. SlotFormer: Unsupervised Visual Dynamics Simulation with Object-Centric Models. ICLR 2023.