Learning Hierarchical Image Segmentation For Recognition and By Recognition

Dear reviewer GfNL,

Thank you for your valuable feedback and comments. We appreciate your remarks on the motivation, method design, and experimental results of our work. We will address your concerns and questions in the response below.

---

[W1] Tone is a bit too strong

Thank you for your suggestions. We revised our manuscript accordingly.

---

[Q1] How to incorporate positional encodings?

We added the positional encodings instead of concatenating them like standard ViTs. Please note that $E_\text{pos}$ is added ($+$) to the tokens in the equation on page 4 of our paper.

---

[Q2] Centroids from position or feature space?

The centroids are sampled based on token similarity in the feature space, though it also reflects the position information through positional encodings. We have clarified this on page 5 of our revised manuscript.

---

[Q3] Reference for linear probing on IN-100

Linear probing is a widely used protocol for evaluating self-supervised learning methods [1-3]. It involves training a linear classifier on top of learned representations to measure classification accuracy. Prior works [4-5] used ImageNet-100 for linear probing, where the top-1 accuracy (~80%) in Table 1 matches that in Table 4 of our paper. The detailed procedure has been explained in Appx. C.4 of our manuscript.

[1] He et al. Momentum Contrast for Unsupervised Visual Representation Learning. CVPR 2020.
[2] Chen et al. A Simple Framework for Contrastive Learning of Visual Representations. ICML 2020.
[3] Caron et al. Emerging Properties in Self-Supervised Vision Transformers. ICCV 2021.
[4] Xiao et al. What Should Not Be Contrastive in Contrastive Learning. ICLR 2021.
[5] Costa at al, Solo-learn: A Library of Self-supervised Methods for Visual Representation Learning. JMLR 2022.

---

Please let us know if you have any further concerns.

Sincerely,
Authors

Dear reviewer hsyp,

Thank you for your valuable feedback and comments. We appreciate your remarks on the motivation, experimental results, and presentation of our work. We will address your concerns and questions in the response below.

---

[W1] Technical novelty over token pooling

Our graph pooling is just a device for implementing our concept of hierarchical segmentation, and any token pooling method can be used. We emphasize that our goal is to develop a framework that integrates segmentation into the recognition process, unlike previous token pooling methods that focus on ViT efficiency. As a reminder, we have discussed this in the related section, properly mentioning both suggested works.

We employed graph pooling because it performed the best in our experiments. In particular, we compared various token pooling algorithms in Table 6a, including Token Pooling (Marin et al., 2021) and FINCH (Sarfraz et al., 2019). Additionally, following your suggestion, we conducted further comparisons with state-of-the-art methods, TCFormer (Zeng et al., 2022) and ToMe (Bolya et al., 2023), using the same setup as in Table 6a. For TCFormer, we modified the Clustering-based Token Merge (CTM) module by removing the convolutional layer to apply it to our superpixel tokens. For ToMe, we reduced the number of tokens per layer to 16 to align with the latency of other methods. We stated the implementation details in Appx. A.5.

The table below presents the results, including both the values from Table 6a and the new results. While both CTM and ToMe perform well, our graph pooling outperforms them. We have updated the table in our revised manuscript.

---

[W2] Ablation study on superpixel and cluster sampling methods

Superpixels. We used SEEDS superpixels for simplicity, as they performed well. However, we note that CAST also performs well with other reasonable superpixel methods. To verify this, we conducted an additional ablation study on superpixel methods. Specifically, we train CAST-S models using SEEDS (Bergh et al., 2012) and SLIC (Achanta et al., 2012) superpixels with MoCo objectives. We report the linear probing accuracy on ImageNet-1K using the models trained on ImageNet-1K and mIoU on Pascal VOC using the models trained on COCO, both before (left) and after (right) fine-tuning.

The table below compares the classification and segmentation performance of SEEDS and SLIC. Both superpixels performed reasonably well, with SEEDS outperforming SLIC by capturing better boundaries. We included deeper discussions and visual examples of superpixels in Appx. B.8 of our revised manuscript.

Nevertheless, we acknowledge that CAST indeed depends on the quality of superpixels. We discussed this limitation in Appx. B.7, illustrating that superpixels may not capture thin structures like light poles. Addressing this issue, perhaps by jointly learning superpixels, would be an interesting future direction.

Cluster sampling. Our cluster sampling is jointly learned with feature development in CAST. On one hand, our cluster sampling and assignments are determined by the learned feature distances. On the other hand, our clusterings facilitate the final recognition process, guiding feature learning. Thus, cluster sampling and feature learning mutually influence each other in CAST. We note that Farthest Point Sampling (FPS) is just one natural method for partitioning features into clusters, and other reasonable clustering methods that encourage maximum margins between clusters would perform similarly well.

---

[Q1] Definition of the MLP ratio?

The MLP ratio refers to the multiplier for the dimension of the MLP used in the self-attention block within our graph pooling module. We clarified this in our revised manuscript.

---

Please let us know if you have any further concerns.

Sincerely,
Authors

Dear reviewer RnRE,

Thank you for your valuable feedback and comments. We appreciate your remarks on the motivation, method design, and experimental results of our work. We will address your concerns and questions in the response below.

---

[W1] Runtime analysis

The computational cost of superpixel generation and graph pooling can be reduced by the decreased number of tokens required for computing self-attention. This advantage becomes more pronounced when using larger models, where self-attention blocks dominate the entire cost. To validate this, we analyze the latency of model inference and superpixel generation. Our system comprises a 32GB Nvidia Titan V GPU card and two Intel(R) Xeon(R) CPU E5-2630 v4 processors, totaling 20 CPU cores. We utilize the PyTorch machine learning framework with 24 workers, a batch size of 64, and an image resolution set to 224x224.

In our system, CAST-B achieves a lower average inference latency of 217 ms compared to ViT-B with 273 ms. SEEDS takes 73 ms to generate superpixels from the batch of images. However, we remark that the current SEEDS implementation is not fully optimized. Employing GPU implementation or parallelizing the process with more CPU cores can alleviate the bottleneck in superpixel generation. Furthermore, the cost of superpixel generation becomes less significant with larger models, which are commonly used in practice. We have included these discussions in Appx. B.1 of our revised manuscript.

---

[W2] Ablation study on superpixel methods

CAST performs well with reasonable superpixel methods. To verify this, we conducted an additional ablation study on superpixel methods. Specifically, we train CAST-S models using SEEDS (Bergh et al., 2012) and SLIC (Achanta et al., 2012) superpixels with MoCo objectives. We report the linear probing accuracy on ImageNet-1K using the models trained on ImageNet-1K and mIoU on Pascal VOC using the models trained on COCO, both before (left) and after (right) fine-tuning.

The table below compares the classification and segmentation performance of SEEDS and SLIC. Both superpixels performed reasonably well, with SEEDS outperforming SLIC by capturing better boundaries. We included deeper discussions and visual examples of superpixels in Appx. B.8 of our revised manuscript.

Nevertheless, we acknowledge that CAST indeed depends on the quality of superpixels. We discussed this limitation in Appx. B.7, illustrating that superpixels may not capture thin structures like light poles. Addressing this issue, perhaps by jointly learning superpixels, would be an interesting future direction.

---

Please let us know if you have any further concerns.

Sincerely,
Authors

Dear reviewer 5FXt,

Thank you for your valuable feedback and comments. We appreciate your comments on the motivation, method design, presentation, and applicability of our work. We will address your concerns and questions in the response below.

---

[W1] Discussion on the role of superpixels

We have shown that superpixels are not the sole reason for improvements. Please refer to the discussion in Table 3a, which reports (non-hierarchical) semantic segmentation performance. Here, superpixels enhance segmentation by providing better boundaries, as you suggested. However, combining hierarchical pooling leads to even greater improvements, enabling the model to grasp the structure of entire objects. This supports that both superpixels and pooling are essential, even for applications that do not require explicit hierarchy. We have further clarified these points on page 8 of our revised manuscript.

We have included the reorganized Table 3a below for your convenience. In this table, we report the mIoU (left) and boundary F-score (right) of segmentation before and after fine-tuning the models. Superpixels improve the original ViT by +5.1% in the boundary F-score. However, hierarchical pooling further enhances ViT + Superpixel by +6.8%.

---

[W2] Pooling over positional embeddings

Pooling over positional embeddings is as natural as pooling over token features, with the former characterizing spatial location and the latter characterizing appearance, which are two aspects of a segment. In fact, averaged positional embeddings serve as summary statistics for representing pooled segments, even when the corresponding location falls outside of a non-convex shaped segment pool. Prior works have also adopted a similar design choice. For example, SLIC (Achanta et al., 2012) calculates centroids by averaging LAB color features and XY locations within pixel clusters.

---

[W3] Top-down recognition during inference

Thank you for the insightful suggestion! As you noted, CAST only considers the top-down pathway during training, guided by the recognition objectives. This knowledge is encoded in the model and reflects the bottom-up pathway during inference. While this enables the model to learn the general top-down elements, the segmentations will not change based on what the top layers predict at inference time.

To extend CAST by incorporating the top-down pathway during inference, we employ test-time adaptation (TTA) [1], specifically TENT [2], with the classifier trained on top of the CAST backbones. We apply TENT to each sample to adapt the model and maximize its prediction confidence. As a result, CAST refines its object segments to align with its initial belief: If this image depicts the predicted class, which parts contribute to this prediction?

We have included the detailed approach and experimental results in Appx. D of our revised manuscript. We trained a dog vs. non-dog classifier on PartImageNet and evaluated the evolution of segmentation before and after TTA. Figure 16 showcases visual examples where TTA improves the segmentation of CAST by capturing object shapes more effectively (e.g., not missing legs in rows 1-3) while reducing attention to unnecessary details (e.g., window frames in row 3). The improvement is more substantial for challenging samples that the CAST originally fails on, increasing the mIoU from 69.5% to 76.6% (+7.1%) for samples with an original mIoU of less than 80%.

[1] Sun et al. Test-Time Training with Self-Supervision for Generalization under Distribution Shifts. ICML 2020.
[2] Want et al. Tent: Fully Test-time Adaptation by Entropy Minimization. ICLR 2021.

---

[Q1] How to generate segments from ViT?

We generate segments from ViT by applying K-Means clustering to the final output tokens. We bilinearly upscale the feature map and then apply K-Means to the pixel-level features to align the segments with the input image resolution. Similar to CAST, we cross-reference clustering assignments across different levels to achieve hierarchical image segmentation. To maintain a consistent cluster hierarchy, we iteratively run K-Means, reducing the number of clusters by grouping the clusters from the previous iteration into coarser clusters. We have clarified this procedure on page 5 and in Appendix A.4 of our revised manuscript.

---

[Q2] PE added to the CLS token?

Following MoCo-v3, we used fixed sine-cosine positional embeddings for patches and an all-zero vector for the CLS token. The all-zero vector matches the length of positional embeddings with tokens but has no impact. We have clarified this in the equation on page 4 of our revised manuscript.

---

Please let us know if you have any further concerns.

Sincerely,
Authors