Revisiting the Integration of Convolution and Attention for Vision Backbone

W1: Introducing the soft clustering and dispatching modules adds complexity to the implementation, which might pose challenges for practical deployment.

The clustering and dispatching modules involve only standard and widely used operators such as Matrix Multiplication and SoftMax, which are supported by all deep learning libraries across different hardwares, such as CPUs and GPUs. Therefore, the complexity of the implementation should not be a problem.

W2: The use of a static number of semantic slots for all images may lead to redundancy and inefficiency in certain cases.

The static number of semantic slots is indeed a limitation of our method, as we mentioned in Section 5. However,

• On the one hand, we have observed that such a simple strategy is already sufficiently efficient to achieve a good efficiency-performance tradeoff, as demonstrated in Figure 2.
• On the other hand, dynamic allocation of the slots still has the limitation on efficiency in the typical batched inference scenarios: an image with fewer slots needs slot padding to align an image with more slots.

Although it may not be easily addressed, it would be interesting for future work to consider the dynamic allocation of slots.

W3: The clustering strategy, though effective, could be further optimized. The current implementation might still be computationally intensive for real-time applications.

• Our clustering strategy is designed to be lightweight and only involves a pooling operation, a matrix multiplication followed by SoftMax. The three operations are widely used and highly efficient in all deep learning libraries.
• The clustering takes less than 5% FLOPs of the whole models. It is far from being a computation bottleneck.

Q1: How sensitive is the overall performance of the GLMix model to the choice of the clustering strategy? Have the authors tested alternative clustering methods, and what were the outcomes?

The design choice of the clustering strategy is significant. According to our ablation study on the clustering strategy in L312-317 and Table 6,

• using k-means clustering not only produces a significantly lower throughput (835.9 im/s -> 440.6 im/s) but also incurs unstable training;
• clustering initialization with per-image adaptive pooling improves the performances over static initialization with learnable parameters (82.5% vs. 82.1%).

Q2: What are the potential limitations of using a static number of semantic slots, and have the authors considered the dynamic allocation of semantic slots based on the complexity of the input image?

We have addressed this issue in W2. Please refer to our reply to W2 above.

Q3: How adaptable is the GLMix integration scheme to different types of vision tasks, such as object detection, instance segmentation, and semantic segmentation? Are there any task-specific adjustments needed for optimal performance?

• Keeping the number of slots consistent (i.e., 64) with image classification is good enough for dense prediction tasks. For object detection (Table 4 top group）, instance segmentation (Table 4 bottom group), and semantic segmentation (Table 5), we do not apply any task-specific adjustments and still obtain state-of-the-art performances.
• We have also tried using more slots in the GLNet-4G backbone for semantic segmentation with UperNet. Using more slots, such as 100 or 256, did not improve the performances (50.6 mIoU -> 50.5 mIOU or 50.6 mIOU).

Q4: What are the scalability limits of the GLMix models when dealing with very high-resolution images or videos?

• With GLMix, memory and computation grow linearly w.r.t. the input size, as the clustering and dispatching take linear complexity, and the attention among slots takes constant complexity. Therefore, the scalability should be similar to linear attention models such as SwinTransformer.
• In practice, the scalability limits depend on memory and latency constraints, as well as engineering optimizations. For reference, below we show the peak memory occupancy and latency w.r.t. different input resolutions and batch sizes. Note that the single-input cases (batch size = 1) are heavily unoptimized in Pytorch and can be further improved with advanced inference engines like Nvidia Triton.

Thanks for your response. We hope our previous reply has addressed your main concerns. Please feel free to reach out if you have any additional questions or require further clarification on any aspects of our work.

W1: Please comment on the effectiveness of GLNet against SMT.

• For classification, the throughput/efficiency of SMT is not as good as its parameters and FLOPs indicate. This is because its core design, scale-aware modulation (SAM), relies heavily on depthwise convolutions (DWConvs), which cannot utilize the GPUs well due to low arithmetic intensity. Note that although we also use DWConvs, we only use it once in each GLMix block.

• For object detection (Table 4) and semantic segmentation (Table 5), GLNet performs better and has lower FLOPs. This implies that GLNet has better scalability to high-resolution inputs than SMT.

W2: What's the intuition of applying self-attention operation on the slots? Is it to ensure that slots do not collapse to the same object?

• The self-attention operation on the slots is for inter-object/inter-region relation learning.
• We have tried removing the self-attention over slots, and the slots do not collapse to the same object, possibly because the column-wise and row-wise SoftMax operations (Eq. 3 and Eq. 4) have introduced both inter-slot and inter-position competitions. However, the IN1k accuracy drops from 82.5% to 82.0% for the GLNet-STL.

W3: Could the authors please explain the intuition behind the dispatching module? Is it because it would be easier to integrate with the features obtained after conv layers?

The goal of the dispatching module is to propagate inter-region/object relation to spatial positions in the feature grid. Indeed, it is crucial for the fusion with the Conv layer processed features, as you point out here.

W4: In Figure 4, the clustering before and after applying the GLMix block are exactly the same. It's not clear what is being learnt by the GLMix block.

The GLMix block performs feature transformation. The input and output of the GLMix block are both feature maps. In Figure 4, we colorize both the input and output feature maps according to the expected clustering so that they look the same, which may be the reason for the confusion. We will revise the figure to be more clear (e.g., by removing the clustering colorization on the input feature map). Thanks for pointing out the issue.

W5: The observation on the effects of varying number of slots differs from DINOSAUR. Could the authors please comment on this difference? Can the authors vary the slots to somewhere between 5-10?

• There are at least two aspects that make our observation on slot numbers differ from DINOSAUR:
  • Supervision signal. DINOSAUR is an unsupervised framework for object discovery while our GLNet is trained under a supervised learning framework. DINOSAUR may require setting a proper number of slots to serve as a prior on how many objects can emerge in an image from their datasets.
  • Evaluation metric. DINOSAUR uses the foreground adjusted rand index (FG-ARI), a no-reference metric for clustering quality, while we use the IN1k accuracy in the ablation study.
• We have tried to use 9 slots (initialized with 3 $\times$ 3 pooling) in GLNet-STL; the IN1k accuracy drops from 82.5% to 81.9%.

Q1: Can the authors please clarify their architecture? What does a single layer look like in GLNet? Do the authors use a pretrained backbone?

The architecture design is specified in Section 3.3. To summarize,

• The macro architecture of GLNet follows most existing hierarchical transformers: it has four stages with gradually smaller spatial resolutions (1/4, 1/8, 1/16, 1/32) and larger hidden dimensions (C, 2C, 4C, 8C). Each layer contains a spatial mixing block (the GLMix block or MHSA block) followed by a channel mixing FFN/MLP. The detailed configurations can be found in Table 2.
• We do not use a pretrained backbone. Since there are architectural differences, it is impossible to inherit the weights from existing architectures. We train the model from scratch on ImageNet and then use the weights to initialize the backbone for downstream dense prediction tasks, following existing works such as SwinTransformer.

Q2: Is there any specific reason apart from the smaller feature resolution that the GLMix block is applied to the 3rd layer?

The GLMix block is applied to the 1st, 2nd, and 3rd stages, and each stage has several layers according to the model size (see Table 2). The 4th stage uses a standard attention block as the resolution (1/32 input size) is sufficiently small so that the standard attention is affordable and beneficial for the performance. See also Section 3.3 for more details.

Q3: There are missing references to hybrid ViTs, which the authors can add and discuss.

GLNet distinguishes itself from existing works, including the recommended references, by applying MHSAs and Convs at different granularity levels, i.e., the object/region level and pixel level. Thanks for bringing these works to our attention. We will reference and discuss them in our revision.

Q4: Could the authors also provide the throughput details for table 3,4 and 5.

• It isn't easy to provide throughputs for these tables because most of the compared works did not report the data.
• For classification (Table 3), Figure 2, together with the SMT throughputs in W1, has covered the latest SOTA models for comparison.
• For object detection (Table 4) and semantic segmentation (Table 5), the throughputs may not be that useful because the task-specific heads, instead of the backbone, consume most computations.

Thanks for the response. Below are our replies to the new comments.

C1: further clarification on the GLMix block.

Your latest comments on the GLMix block are mostly correct. However, we would like to clarify one thing. Although the clustering and dispatching modules are similar to cross attention, the two modules are designed to be lightweight so that they have no QKV projections, no multihead mechanism, and the two modules share the correspondence/assignment logits.

C2: the slots are random features during early epochs. Does this affect the learning?

We did not encounter any difficulties caused by the random slots in early epochs. Previous Perceiver/DETR-like architectures, which involve something similar to the slots, also have no problems here. One possible reason for this phenomenon is that even random projection can preserve distances/similarities well.

C3: request for the visualization of non-object-centric images from IN1k, of semantic slots evolution against training epochs, and of semantic slots with Perceiver-like non-image-conditioned design.

• Unfortunately, while authors could optionally upload a PDF in the global response section during the initial rebuttal phase, we are not allowed to edit this section now. Therefore, we are unable to submit a PDF now.
• By looking at the visualization of non-object-centric images, we still observe meaningful semantic grouping effects of the things and stuff.
• Unfortunately, we did not keep the model snapshots at different epochs or for the Perceiver-like non-image-conditioned design, which are required to visualize the latter two items. We will redo the experiments and will describe the results as soon as possible. However, since we are attending the ACL conference now, it may take a couple of days for us to get back to you, but we will try our best to reply soon.

We have done some further visualizations as per your request. Below are our observations.

• At the end of the 1st epoch, a few semantic slots can already be associated with foreground objects, while the others are either uniform or scattered patterns. At the end of the 5th epoch, the semantic grouping effect becomes similar to Figure 5 in the paper. In subsequent epochs (e.g., 10th, 50th, ...), the grouping effect gradually becomes more obvious.
• With non-image-conditioned initialization for the slots, while a similar semantic grouping effect is observed, we find that nearly all slots are associated with the foreground objects in many cases. This possibly accounts for the degraded classification accuracy with such a design (82.5% -> 82.1%, Table 6): the background information is ignored, even though it may be helpful in some cases.

W1: IN-1k size dataset cannot evaluate "scalability." It is advisable to revise this argument.

We thank this reviewer for the suggestion. However, in Line 6, we actually refer to the scalability w.r.t. the input size, instead of the dataset size. This can be reflected by the FLOPs-performance tradeoff on dense prediction tasks such as semantic segmentation (Table 5) and object detection (Table 4), where high-resolution inputs are used. We apologize for the confusion and will clarify it in our revision: "... the scalability issue w.r.t. the input resolution for vision transformers".

W2: Clarify the novelty and contribution in comparison with existing clustering attention works.

• As an integration scheme of attention and convolution, GLMix distinguishes itself from existing works in using Convs and MHSAs at different granularity levels, i.e., Convs on fine-grained feature maps and MHSAs on coarse semantic slots. We have found that with Convs applied on the feature grid, MHSAs can be aggressively applied to a few semantic slots while achieving comparable and even better performances than existing state-of-the-arts.
• Unlike ClusTR [1] and TCFormer [4], which use DPC-KNN for the clustering, the soft clustering module in our work is fully learnable and does not rely on predefined rules. In comparison with ClusterFormer [3] and PaCaViT [4], which perform cross-attention between the feature grid and cluster representations/slots, our work performs self-attention over the slots (i.e., queries and key-value pairs are both from the slots), making the attention even more lightweight. We will incorporate this discussion in the Related Work section in our revision.