Bootstrapping Top-down Information for Self-modulating Slot Attention

Sensitivity to codebook size

We acknowledge the performance dependence on the codebook size, as noted in our limitations section. However, we would like to emphasize that the optimal codebook size was determined automatically using the training set only (without the validation set). To choose the codebook size, we monitor the perplexity of the code usage distribution during training. Perplexity—the exponent of entropy—indicates how uniformly the codes are being used and typically increases as the codebook becomes larger. However, when the codebook size exceeds the number of semantic patterns in the data, some codes become neglected, causing low code utilization [D, 45, 26]. Hence, the ideal codebook size leads to the maximum perplexity.

To find the optimal codebook size, we start with a size of 64 and monitor the perplexity of the codebook over training. After sufficient iteration (e.g., 250K), the codebook size is doubled repeatedly until the perplexity reaches a plateau. For example, on COCO, the perplexity when the codebook size is 256, 512, and 1024 are 176.9, 253.9, and 242.8, respectively, where 512 was chosen as the final size. Thereby, an adequate codebook size could be efficiently determined without using the validation set.

Experiment on models other than DINOSAUR

We appreciate your suggestion and have implemented our self-modulation technique with the original Slot Attention setting [28], which trains the encoder from scratch and uses an image reconstruction objective. The following table summarizes the results on the CLEVR6 dataset:

We observe a significant improvement in mBO, showing that our self-modulation technique is applicable to slot attention and provides complementary benefits. Although FG-ARI decreased, mBO is considered more robust and should be prioritized when evaluating model performance, as detailed in our response to reviewer 3ckb ("Necessity & effect of VQ: (3) Difference in FG-ARI and mBO in VQ Ablation"). We have also included qualitative results in Figure R3 of the attached PDF, which demonstrates that self-modulation markedly improves segmentation. These results indicate our method's effectiveness with different encoder configurations and training objectives, suggesting it does not heavily rely on pretrained features. We will add the results and discussion in the revision.

Does the codebook also contain positional information?

Our codebook primarily encodes semantic content, largely independent of the object position (Figure 2 of the main paper). We hypothesize that this is because semantic information dominates slot representations, with positional information occupying a small subspace. This is supported by $K$-means clustering of DINOSAUR slot representations, which automatically groups semantic categories using simple L2 distance (Figure R6 of the attached PDF). While our approach empirically extracts mainly semantic information, explicitly disentangling positional and semantic information remains an interesting future direction, with work like [3] offering promising leads.

Inconsistent numbers between tables

We apologize for the confusion. Our mBO$_i$ on COCO of 33.3 in Table 3 is a typo, and the correct number is 33.0. We will revise the paper to include the correct measurement. Despite this error, our method still outperforms the LSD [18] by 2.6%p, showing the effectiveness of our approach.

Additionally, as mentioned in line 234 on page 6, the in-depth analysis and ablation studies (Tables 4, 5, and 6) were conducted with models trained for 200K iterations, rather than the full 500K iterations, due to computational constraints. This explains the differences between Tables 1-3 and Tables 4-6. We found that the models mostly converged at 200K iterations and thus was enough to reflect the performance trends of models trained for the full 500K iterations. To improve clarity, we will update the captions of Tables 4-6. We appreciate your attention to these details.

Can spatial modulation be called “top-down”?

We appreciate the reviewer’s comment on our top-down approach. We respectfully clarify our reasons for calling spatial modulation a top-down approach.

A top-down approach is defined by the use of prior knowledge (contextual or task-relevant) to guide visual processing, while a bottom-up approach relies solely on sensory data. As the reviewer noted, our semantic modulation using vector quantization aligns with this definition, leveraging contextual knowledge learned from the dataset.

Although the spatial information is derived from the same image, the spatial modulation provides rough object saliency maps to the subsequent slot attention, which is crucial task-relevant information for object discovery. It is task-relevant because it explicitly guides the model to discover objects within the provided saliency map, narrowing the search space. Such spatial cues are recognized as a form of top-down knowledge in human vision research [A, B, C], guiding attention to regions likely containing objects of interest.

In conclusion, while our semantic and spatial modulations differ in their information sources, both embody top-down processing principles by providing crucial task-relevant guidance to the visual processing pipeline.

---

[A] SUN: Top-down saliency using natural statistics, Visual cognition, 2009

[B] Components of visual orienting, Attention and performance X: Control of language processes, 1984

[C] Oculomotor strategies for the direction of gaze tested with a real-world activity, Vision research, 2003

[D] Straightening Out the Straight-Through Estimator: Overcoming Optimization Challenges in Vector Quantized Networks, ICML, 2023

Clarification on term “top-down pathway” / modulating encoder

We appreciate the reviewer's comment on our use of the terminology, "top-down pathway." We respectfully explain that our terminology is appropriate for the following reasons: In our context, "top-down" refers to using higher-level task-relevant information to guide lower-level processing. Our method achieves this by: a) Extracting semantic information via vector quantization to enforce the focusing on certain features. b) Using spatial information from the previous slot attention to guide object localization. Both of the above mechanisms bootstrap top-down information beyond pure bottom-up processing, without requiring additional supervision. While our top-down pathway is confined to the slot attention, it still introduces a meaningful top-down influence on the features by modulating values in slot attention. We agree that extending the top-down modulation to the encoder is an interesting direction for future research. However, our current approach strikes a balance between effectiveness and efficiency, as modulating the encoder would significantly increase computational cost (the encoder accounts for over 71% of total compute). Our method achieves meaningful improvement without this additional overhead. For a detailed discussion of our method's efficiency, please refer to our response to Reviewer 3ckb (“Computation cost analysis”).

Value modulation vs. slot initialization

We appreciate the insightful suggestion. We did consider such a variant that initializes the slots with the quantized representations from the previous slot attention. However, this approach did not perform well in our experiments. Given that slot attention consists of 3 iterations, initializing the second slot attention with quantized representations from the first is equivalent to inserting VQ in the middle of 6 iterations. To evaluate this, we insert VQ into DINOSAUR's slot attention with 6 iterations, after the third iteration. Results on COCO with 200K training schedule are presented below, together with the results presented in the Table 5 of the main paper:

Introducing VQ to DINOSAUR with 6 iterations leads to a performance decrease of 11.1%p and 15.3%p in FG-ARI and mBO, respectively (row 1 vs. 3 of the table). The severe performance degradation is likely due to the disruption in slot attention's convergence process. As shown in [6], slot representations naturally converge over its recurrent iterations. Discretizing in mid-process introduces a sudden shift in the slot representation, which will harm the overall optimization and convergence. Our method avoids this issue by modulating inner value activations, which can modulate the recurrent slot update process without directly changing the slot representation. By doing so, we can exploit the top-down information without hindering convergence of the slot attention.

Comparison to CutLER

We appreciate the suggestion to compare with CutLER [A], a recent approach to unsupervised object segmentation. While CutLER's full pipeline includes self-training of Mask R-CNN, we focus on comparing our method to CutLER's MaskCut algorithm for pseudo-mask generation, as this is the most direct comparison to our approach. Our method’s predictions can also be used as pseudo labels for training Mask R-CNN. We reproduced MaskCut using the official CutLER repository and evaluated it on the COCO dataset:

Our method significantly outperforms MaskCut across all metrics, demonstrating its effectiveness for object discovery. We will add these results and discussion in the revision.

Dependence on the pretrained feature encoder

We appreciate the insightful suggestion. Regarding the inquiry, we conducted an experiment with the settings of the original slot attention [28], which trains the encoder from scratch (response to reviewer Yith, PJTV: “Experiment on models other than DINOSAUR”). Interestingly we observe substantial performance gain even in this setting, demonstrating that our method's improvements does not solely arise from using a pretrained feature encoder. For more detailed results and discussion, please refer to the corresponding response. We will discuss this experiment in the future revision as well.

---

[A] Cut and Learn for Unsupervised Object Detection and Instance Segmentation, CVPR, 2023

Experiment on models other than DINOSAUR

We appreciate your suggestion for additional experiments. We have implemented our self-modulation technique with the original slot attention setting [28], which includes training encoders from scratch and using an image reconstruction objective. The following table summarizes the results on the CLEVR6 dataset:

We observe a significant improvement in mBO, showing that our self-modulation technique is applicable to slot attention and provides complementary benefits. Although FG-ARI decreased, mBO is considered more robust and should be prioritized when evaluating model performance, as detailed in our response to reviewer 3ckb ("Necessity & effect of VQ: (3) Difference in FG-ARI and mBO in VQ Ablation"). We have also included qualitative results in Figure R3 of the attached PDF, which demonstrates that self-modulation markedly improves segmentation. These results indicate our method's effectiveness with different encoder configurations and training objectives, suggesting it does not heavily rely on pretrained features. We will add the results and discussion in the revision.

Slots reassigned to other object after modulation

Thank you for your acute observation. We sincerely apologize for the discrepancy in Figures 3 of the main paper and B.5 of the appendix, which resulted from inadvertently pairing slot attention maps from different samples: the slot attention maps after self-modulation are correct but attention maps before self-modulation were wrongly paired. We have provided the updated figures as Figure R1 and R5 in the attached PDF.

As the reviewer correctly pointed out, slots are modulated based on the information captured during the first slot attention. In the corrected Figure 3 (Figure R1 of the attached PDF), row 3, the 5th slot, which initially captures the skater with less certainty, is modulated to capture the skater with higher confidence by leveraging the semantic and spatial information from the initial slot.

We would like to reaffirm that our original observation—“ modulation dynamically refines the attention maps, depending on how well they have captured the scene”—remains valid. For instance, in the 2nd row, the 1st slot initially captures both the plates and the orange, but after modulation, it is associated exclusively with the orange, yielding the plates to the 7th slot which more accurately captures the plates during the first slot attention. Similarly, in the 4th sample, the 3rd slot, which initially associates with the straw in a trivial manner, is modulated to identify and locate the straw more accurately. On the other hand, the attention maps of the 1st row are refined at the boundaries only since the initial attention maps were well structured.

Once again, we apologize for any confusion caused by this error. We appreciate your understanding and the opportunity to clarify our findings.

Does the codebook also contain positional information?

While slot representations can indeed contain positional information due to the reconstruction objective, our observations suggest that the codebook primarily encodes semantic content. Performing $K$-means clustering on slot representations from DINOSAUR, we observed that the clusters are primarily grouped by semantic categories rather than spatial positions, as shown in Figure R6 of the attached PDF. This indicates that semantic information dominates in slot representations. Figure 2 in the main paper further supports this, showing objects with similar semantics but different positions mapped to the same code. Thus, while present, positional information likely occupies a small subspace in slot representations, with semantic content being the primary factor captured by our codebook.

Question about codebook visualization/semantics

Figure 2 is not heavily cherry-picked. Most codes consistently represent a single semantic concept, with a few codes that portray supercategories such as animals and humans (people and their hands), as shown in Figure R4 of the attached PDF.

Also, we did notice an interesting edge case where some codes capture only the top-left-most patch, as shown in Figure R2 of the attached PDF. We attribute this to the autoregressive decoding process. During decoding, the top-left patch must be reconstructed first without any context from previous patches. Consequently, in images with less visual complexity, some slots appear dedicated to capturing these top-left patches. However, these special codes are minimal (1 or 2 out of 512 codes) and have limited impact. We will include more codebook visualizations and a detailed discussion in the revision.

Do modulated slots perform semantic grouping more than instance grouping?

Thank you for the insightful observation. To quantitatively assess whether our method favors semantic grouping over instance grouping compared to DINOSAUR, we conducted an analysis using two metrics: Instance Precision: The precision score with instance-level GTs Semantic Recall: The recall score with semantic segmentation GTs If our model was biased toward semantic grouping, we would expect to see a lower instance precision (as semantic segmentation ground truth masks often have larger sizes) and a higher semantic recall compared to DINOSAUR. However, our analysis on the COCO dataset shows that our method improves both metrics, as shown in the table below.

These results suggest that the proposed top-down pathway enhances identifying both individual instances and semantic categories rather than favoring one.

Validity of mIoU scores in MOVI-C/E

Thank you for your astute feedback. We sincerely apologize for the mistake in the mIoU calculations on the MOVI datasets and any confusion it may have caused. As you correctly pointed out, the mBO should be greater than or equal to the mIoU. The reported mIoU is inflated due to omitting the first ground-truth mask. We have provided the corrected mIoUs in Table R1 of the attached PDF. Importantly, despite this error, our revised results still demonstrate that our method outperforms DINOSAUR, substantiating our original claim about the benefits of our top-down pathway. We greatly appreciate your diligence in identifying this issue and allowing us to correct it.

Necessity & effect of VQ

Thank you for your insightful comment regarding Vector Quantization (VQ). We explain the necessity of VQ by addressing the inquiries below:

(1) Necessity of VQ for Self-Modulation

VQ is fundamental to our method since it allows the model to capture recurring semantic patterns from rough object representations produced by the first slot attention. Similar to online clustering, VQ maps a continuous slot representation to the nearest discrete code. Throughout this process, codes in the codebook are updated to encode only the semantic information from the noisy object representations; this is consistent with clustering being used to capture high-level semantic knowledge from noisy inputs in various tasks [D, 44]. Therefore, incorporating VQ into the top-down pathway enables our model to focus on core semantic concepts of the dataset, which are bootstrapped from noisy representations without extra supervision. This leads to more refined object-centric representations.

(2) Sensitivity to Codebook Size

We acknowledge the performance dependence on the codebook size, as noted in our limitations section. This characteristic is indeed common in clustering-based methods. However, we would like to emphasize that the optimal codebook size was determined automatically using the training set only (without the validation set). To choose the codebook size, we monitor the perplexity of the code usage distribution during training. Perplexity—the exponent of entropy—indicates how uniformly the codes are being used and typically increases as the codebook becomes larger. However, when the codebook size exceeds the number of semantic patterns in the data, some codes become neglected, causing low code utilization [E, 45, 26]. Hence, the ideal codebook size leads to the maximum perplexity. To find the optimal codebook size, we start with a size of 64 and monitor the perplexity of the codebook over training. After sufficient iteration (e.g., 250K), the codebook size is doubled repeatedly until the perplexity reaches a plateau. For example, on COCO, the perplexity when the codebook size is 256, 512, and 1024 are 176.9, 253.9, and 242.8, respectively, where 512 was chosen as the final size. Thereby, an adequate codebook size could be efficiently determined without using the validation set.

(3) Difference in FG-ARI and mBO in VQ Ablation

While we agree that mBO is the more reliable metric, FG-ARI can provide complementary information when mBOs are similar. FG-ARI is considered deceptive because it does not penalize under-segmentations that include background pixels [A], sometimes resulting in misguidedly high FG-ARIs despite low mBOs. However, in Table 6 of the main paper, VQ improves both metrics. The increased mBO, which accounts for background pixels, suggests that the FG-ARI increase is not due to under-segmentation but due to improved foreground object segmentation. Thus, we respectfully claim that the increase in FG-ARI with VQ should be considered as valid evidence for the effectiveness of VQ, especially considering the slight improvement in mBO.

(4) VQ Ablation with Full Training Schedule

We further validate the effectiveness of VQ with an experiment under a full training schedule (500K iterations) on COCO as suggested. The results demonstrate consistent improvement in both FG-ARI and mBO, clearly showing that VQ plays a crucial role in our model:

In summary, while we acknowledge the sensitivity to codebook size, we believe the benefits of VQ outweigh this limitation. Additionally, an appropriate codebook size can be efficiently determined by monitoring perplexity during training.

Computation cost analysis

While our model requires one more forward pass for slot attention, the additional computation cost is negligible. Slot attention accounts for only 0.64% of the total FLOPs, compared to 71.26% for the encoder and 28.10% for the decoder. Thus, our model requires 47.62 GFLOPs while DINOSAUR needs 47.32 GFLOPs. Also, processing the entire COCO 2017 val split takes 71.4 seconds for our model, while 70.5 seconds in the DINOSAUR.

Missing comparison with recent work

Thank you for pointing out the missing related work. We will add comparisons with Rotating Features [C] and SPOT [B] in the revision. We would like to kindly remind the reviewer that SPOT was published at CVPR 2024, after our submission deadline. Moreover, SPOT's self-training and sequence permutation techniques are completely orthogonal to our work on top-down pathways, suggesting potential for future integration of these complementary approaches.

---

[A] Genesis: Generative Scene Inference and Sampling with Object-centric Latent Representations, ICLR, 2020.

[B] SPOT: Self-Training with Patch-Order Permutation for Object-Centric Learning with Autoregressive Transformers, CVPR, 2024

[C] Rotating Features for Object Discovery, NeurIPS, 2023

[D] Deep Clustering for Unsupervised Learning of Visual Features, ECCV, 2018

[E] Straightening Out the Straight-Through Estimator: Overcoming Optimization Challenges in Vector Quantized Networks, ICML, 2023