Memory Efficient Transformer Adapter for Dense Predictions

We appreciate your acknowledgment of our work in reducing memory costs while achieving enhanced performance through extensive experimental evaluations. The following is our response with some necessary explanations. If you have any further questions, please feel free to let us know.

Q1: A detailed breakdown of inference time.

A1: Thanks for your suggestion. The inference time analysis for each component of META is provided in Table 4 of our main paper, and the comparative evaluation of the inference time with previous methods is included in Table S3 of the supplementary materials. We use the frames per second (FPS) during the inference process as the metric to measure the model's inference time.

From Table 4, we can observe that: a) The addition of the Attention branch and FFN branch does not significantly reduce FPS compared to the baseline model, indicating that these two branches do not substantially impede the model's inference time. b) The incorporation of the Conv branch leads to a slight decrease in FPS, suggesting that convolutional layers impact the model's speed due to the increased model complexity. c) The implementation of the cascaded strategy can further enhance the model's inference speed, as this manner allows for the parallel processing of different features, particularly with current GPUs support for parallel computation. Furthermore, the cascaded strategy can facilitate the rapid feature fusion, thus avoiding complex feature interactions and enhancing the inference efficiency.

In Table S3 of the supplementary materials, we compare META with previous efficient attention methods, including Window Attention[R1], Pale Attention[R2], Dense Attention[R3], CSWindow[R4], SimpleAttention[R5], and Deformable Attention[R6]. We evaluate the model's accuracy, complexity, memory, and inference efficiency. The experimental results indicate that our method achieves state-of-the-art performance in both accuracy and inference time.

References:

[R1] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV, 2021.

[R2] Sitong Wu, Tianyi Wu, Haoru Tan, and Guodong Guo. Pale transformer: A general vision transformer backbone with pale-shaped attention. In AAAI, 2022.

[R3] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurIPS, 2017.

[R4] Xiaoyi Dong, Jianmin Bao, Dongdong Chen, Weiming Zhang, Nenghai Yu, Lu Yuan, Dong Chen, and Baining Guo. Cswin transformer: A general vision transformer backbone with cross-shaped windows. In CVPR, 2022.

[R5] Bobby He and Thomas Hofmann. Simplifying transformer blocks. In arXiv, 2023.

[R6] Zhuofan Xia, Xuran Pan, Shiji Song, Li Erran Li, and Gao Huang. Vision transformer with deformable attention. In CVPR, 2022.

Q2: The convolution, layer normalization, and attention may result in multiple tensor reshaping operations.

A2: Yes, the convolution operation, layer normalization, and attention operation may lead to multiple tensor reshaping operations, which is why the model complexity of current vision adapter-based methods is always higher than that of the baseline models. However, these operations are not only essential but also inherently present in the existing vision adapter block, which also serves as the main motivation for our work. Therefore, under the current framework, we have made enhancements for both layer normalization and attention to improving the model's memory efficiency, including sharing layer normalization across multiple modules and substituting standard self-attention with cross-shaped self-attention.

Q3: A thorough analysis of memory costs associated with each operation in META and previous approaches.

A3: In Table 4 of the main paper, we provide experimental results for the effectiveness of each component of META under the evaluation of the model's accuracy, complexity, memory, and inference efficiency. From this table, we can observe that the memory costs associated with META predominantly arise from the Attn branch, which contributes an additional 7.5 GB of memory consumption on ViT-B. In contrast, the incorporation of the FFN branch and the Conv branch does not result in a significant increase in memory usage. The implementation of the cascade strategy necessitates the pre-storage of features across different head layers, leading to an additional memory increase of 0.6 GB on ViT-B.

In Table S3 of the supplementary materials, we compare the proposed META with previous efficient attention methods, including Window Attention[R1], Pale Attention[R2], Dense Attention[R3], CSWindow[R4], SimpleAttention[R5], and Deformable Attention[R6]. From the obtained results presented in the final column of this table, it is evident that our model maintains the lowest memory costs among the compared approaches.

Q4: An additional ablation study without the Attn Branch.

A4: Thanks for this suggestion. During the experimental design phase, we have considered experimental settings that involved using only the FFN branch. In our revised Table 4, we included the ablation results on instance segmentation, object detection and semantic segmentation tasks under this setting. It is observed that while utilizing only the FFN branch results in minimal model complexity and memory costs, there is a significant accuracy decline.

Q5: Compare META with other methods on V100.

A5: In our submission, the reported results are measured by A100 GPUs with per-GPU batch size 2. To avoid any ambiguity, we have added a description of the GPU settings for the inference phase in Section 4.1 of the revised version. The details are as: “The reported inference results are measured by A100 GPUs with per-GPU batch size 2.”

Following your suggestion, we conducted an extra inference on V100 GPUs with 32 GB and a per-GPU batch size of 1. We chose Cascade Mask R-CNN for instance segmentation and object detection as the baseline, ViT-B is used as the backbone. The inference results are presented below.

We can observe that the inference speed is somewhat slower on V100 GPUs. Fortunately, compared with other methods, our method still demonstrates superior performance.

Q6: Details regarding the experimental setup In Table S3.

A6: Following the same settings as in [R7], the attention mechanism is utilized as the ViT-adapter layer. Therefore, during the experimental comparisons, we replace the attention mechanism in the ViT-adapter model with other attention mechanisms to ensure a fair comparison. We have included the detailed settings in Section S5 of the supplementary materials.

Other minor comments:

Q1: About the spatial prior module.

A1: Yes, the used spatial prior module follows the same as the one in [R7]. In Section 3.1 of our revision, we have highlighted this point.

Q2: DeiT in Line 96.

A2: Thanks for this suggestion. We have incorporated DeiT into this section.

Q3: About the ambiguously “which” in Line 182.

A3: Thank you for pointing this out. To eliminate any ambiguity, we have revised the sentence as: "The MEA block is designed to facilitate the interaction between the features extracted from the ViT backbone and the spatial prior module. Our block consists of the attention (i.e., Atte) branch, the feed-forward network (i.e., FFN) branch, and the lightweight convolutional (i.e., Conv) branch."

Q4: About the abbreviation of “Concat” and ” Attn”.

A4: Thanks for this suggestion. Based on your suggestion, we have modified "Cat" to "Concat" in Equations 1 and 3, and changed "Atte" to "Attn."

Q5: Line 223: Should "respectively" be replaced with "sequentially"?

A5: We agree that "sequentially" is a more appropriate choice and have implemented the suggested fix accordingly.

Q6: The use of "common" may introduce ambiguity on Line 166 of Table S2.

A6: Thank you for pointing this out. In this context, "common" refers to the classic normalization method, which is non-shared. To avoid ambiguity, we have revised this to "Non-shared normalization," thereby contrasting it with our proposed shared normalization. This modification enhances the clarity of the table's content.

Thank you for your detailed responses, which have partially addressed my concern (Q3-Q6). However, some concerns remain:

A1: Thanks for your suggestion... model's inference time.

Table 4 presents only the overall fps when ablating different components. I am interested in understanding how many seconds each component consumes during a single inference of your model. For instance, consider providing information similar to the following:

c) The implementation of the cascaded strategy can further enhance the model's inference speed, as this manner allows for the parallel processing of different features

Based on Figure 2 and Section 3.3, the cascaded strategy seems to process each head sequentially rather than in parallel.

A2: Yes, the convolution operation, layer normalization, and attention operation may ... with cross-shaped self-attention.

The MEA block in your method mixes convolutions, layer normalization, and attention, necessitating tensor reshaping operations. Compared to the existing vision adapter block, how many reshaping operations does your method reduce? Could you provide a detailed comparison of the number of reshaping operations required in one MEA block versus those in other adapter blocks?

A1: How many seconds each component consumes during a single inference of your model.

Q1: Thanks for this valuable suggestion. Following your suggestion, we conducted a comprehensive evaluation of the inference time and memory costs for each component during the inference process. META-B with Mask R-CNN under the 3$\times$ training schedule for instance segmentation and object detection is used as the baseline. The results are presented in the following table, where "Others" denotes the cascade strategy employed in our method, along with additional essential operations required for the execution of the ViT adapter, including convolutional layers, feature separation, feature concatenation, and feature addition operations. The rationale behind this representation style is that the cascade strategy cannot be executed independently; it necessitates integration with operations involving other features. From this table, we can observe that the time and memory costs during the inference process of our method primarily stems from the Attn. branch.

A2: The cascaded strategy seems to process each head sequentially rather than in parallel.

Q2: Sorry for the confusion. Yes, the cascade strategy processes each head sequentially. However, it can still facilitate the parallel processing of ViT models by allowing certain operations to be executed concurrently. While the strategy is organized in a cascade manner, each stage can independently and simultaneously process different heads, thereby reducing overall computation time. This strategy effectively utilizes GPU computational resources by overlapping the execution of different stages, resulting in enhanced efficiency in inference, despite the serial nature of the cascaded architecture.

A3: How many reshaping operations does your method reduce? Could you provide a detailed comparison of the number of reshaping operations required in one MEA block versus those in other adapter blocks?

Q3: Thanks for your insightful question. In our MEA block, we employ the cross-shaped attention mechanism aimed at minimizing reshaping operations. Compared to the attention mechanisms employed in other adapter blocks, our approach reduces the demand for reshaping by one instance per execution of a single-head attention. This reduction not only mitigates computational complexity but also decreases memory overhead, thereby enhancing the overall efficiency of the model.

To elucidate this, consider an input tensor with dimensions H $\times$ W $\times$ C, where H denotes height, W denotes width, and C denotes the channel size. In a conventional attention mechanism, the process typically involves three steps, which entail two reshaping operations:

1. The first reshaping: reshaping the input tensor into a matrix of dimensions (H $\times$ W) $\times$ C.

2. Computing the attention matrix, generally of size (H$\times$ W) $\times$ (H $\times$ W).

3. The second reshaping: performing a linear transformation using the attention matrix, which necessitates reshaping back to the dimensions of (H $\times$ W) $\times$ C.

In contrast, the cross-shaped self-attention mechanism requires only a single reshaping operation. Our method operates directly on the original tensor dimensions of H $\times$ W $\times$ C, allowing the resulting attention matrix to be reshaped solely back to H $\times$ W $\times$ C, thereby necessitating only one reshaping step.

## Function MEA_Block(F_sp, F_vit)
1. **Input**: 
   - F_sp: Features from SPM
   - F_vit: Features from ViT

2. **Shared Layer Normalization**:
   - F_sp_ln = LayerNorm(F_sp)
   - F_vit_ln = LayerNorm(F_vit)

3. **Attention Branch**:
   - A_H = Cross_Shaped_Self_Attention_Horizontal(F_sp_ln, F_vit_ln)
   - A_V = Cross_Shaped_Self_Attention_Vertical(F_sp_ln, F_vit_ln)
   - A = Concat(A_H, A_V)

4. **Feed-Forward Network (FFN) Branch**:
   - F_temp = Concat(F_sp, F_vit)
   - F_temp = Conv3x3(F_temp)
   - F_temp_ln = LayerNorm(F_temp)
   - F_ffn = MLP(F_temp_ln)p

5. **Convolutional Branch**:
   - C = Concat(F_sp, F_vit)
   - C = DC(C)  # Depth-wise Convolution
   - C = GLU(C)
   - C = DC(C)
   - C = GLU(C)

6. **Output**:
   - F_out = Conv3x3(Concat(A, F_ffn, C, F_sp, F_vit))

7. **Return**:
   - Return F_out

function cross_shaped_window_attention(x, num_heads, window_size):
    # x: given feature
    # num_heads: head number
    # window_size: window size

    # Get dimensions
    (batch_size, seq_length, d_model) = shape(x)

    # Split into multiple heads
    Q, K, V = split_heads(x, num_heads)

    # Initialize attention output
    attention_output = zeros(batch_size, seq_length, d_model)

    # Initialize previous head's output for cascaded attention
    previous_Q = zeros(batch_size, seq_length, d_model)
    previous_K = zeros(batch_size, seq_length, d_model)
    previous_V = zeros(batch_size, seq_length, d_model)

    # Calculate attention for each head
    for head in range(num_heads):
        for position in range(seq_length):
            # Get cross-shaped window indices
            window_indices = get_cross_shaped_window_indices(position, window_size)

            # Gather Q, K, V for the current window
            if head == 0:
                Q_window = gather(Q[head], window_indices)
                K_window = gather(K[head], window_indices)
                V_window = gather(V[head], window_indices)
            else:
                Q_window = gather(Q[head], window_indices) + previous_Q
                K_window = gather(K[head], window_indices) + previous_K
                V_window = gather(V[head], window_indices) + previous_V

            # Calculate attention scores
            attention_scores = softmax(Q_window * K_window^T / sqrt(d_k))

            # Compute the attention output for the current position
            attention_output[position] = attention_scores * V_window

        # Update previous head's output for the next head
        previous_Q = gather(Q[head], window_indices)
        previous_K = gather(K[head], window_indices)
        previous_V = gather(V[head], window_indices)

    # Final linear transformation
    attention_output = linear_transform(attention_output)
    return attention_output

function feed_forward_network(x):
    # Feed Forward Network
    x = ReLU(linear(x))
    x = linear(x)
    return x

def img2windows(img, H_sp, W_sp):
    B, C, H, W = img.shape
    img_reshape = img.view(B, C, H // H_sp, H_sp, W // W_sp, W_sp)
    img_perm = img_reshape.permute(0, 2, 4, 3, 5, 1).contiguous().reshape(-1, H_sp* W_sp, C)
    return img_perm

class CSWinTransformer:
    def __init__(self, stripe_width=(7, 7), ...):
        self.stripe_width = stripe_width
        ...

    def cross_shaped_window_attention(self, x):
        # Use self.stripe_width to determine the size of the attention windows
        ...

 (batch_size, seq_length, d_model) = shape(x)
 for position in range(seq_length):

We extend our sincere gratitude for your confirmation of our work as simple and fast, addressing the critical yet underexplored issue of memory inefficiency, demonstrating state-of-the-art performance in terms of accuracy and memory usage, and a clear structure with detailed architectural descriptions and clear explanations. The following is our response with some necessary explanations. If you have any further questions, please feel free to let us know.

Q1: Comparison and discussion between CSA and other efficient attention mechanisms.

A1: In Table S3 of the supplementary materials, we have compared META under CSA with other efficient attention methods, including Window Attention[R1], Pale Attention[R2], Dense Attention[R3], CSWindow[R4], SimpleAttention[R5], and Deformable Attention[R6]. All methods are evaluated using their default configurations and the same settings as the adapter injector and extractor in the ViT-adapter model[R7] to ensure fairness. As stated in Section S5 of the supplementary materials, the experimental results demonstrate that our method can achieve state-of-the-art performance in terms of both accuracy and efficiency.

References:

[R1] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV, 2021.

[R2] Sitong Wu, Tianyi Wu, Haoru Tan, and Guodong Guo. Pale transformer: A general vision transformer backbone with pale-shaped attention. In AAAI, 2022.

[R3] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurIPS, 2017.

[R4] Xiaoyi Dong, Jianmin Bao, Dongdong Chen, Weiming Zhang, Nenghai Yu, Lu Yuan, Dong Chen, and Baining Guo. Cswin transformer: A general vision transformer backbone with cross-shaped windows. In CVPR, 2022.

[R5] Bobby He and Thomas Hofmann. Simplifying transformer blocks. In arXiv, 2023.

[R6] Zhuofan Xia, Xuran Pan, Shiji Song, Li Erran Li, and Gao Huang. Vision transformer with deformable attention. In CVPR, 2022.

[R7] Zhe Chen, Yuchen Duan, Wenhai Wang, Junjun He, Tong Lu, Jifeng Dai, and Yu Qiao. Vision transformer adapter for dense predictions. In ICLR, 2023.

Q2: More ablation analysis on object detection and semantic segmentation.

A2: Thanks for your suggestion. In Table 4 of our revised submission, we have included the results of the ablation study experiments on object detection and semantic segmentation. Besides, in Section 4.4 “ablation analysis”, we have added an analysis of the experimental settings and results pertaining to this Table.

Q3: Inconsistency presentation between Formula (1) and part (a) of Figure 2.

A3: Thank you for pointing this out. In our revision, we have fixed Formula 1 by adding the channel concatenation part for F^{sp} and F^{vit} to ensure consistency between Figure 2(a) and Formula 1.

Q4: Visualization results.

A4: The visualized result comparisons are provided in Figures S1 and S2 of the supplementary materials. Specifically, a) In Figure S1, we compare the class activation map for instance segmentation before and after the addition of the Conv branch. From this figure, it is evident that after incorporating the Conv branch, our method focuses more on specific object areas (e.g., "the dog" and "the person") rather than the surrounding regions that may extend beyond the objects themselves. This indicates that our method effectively learns local inductive biases following the integration of the Conv branch. b) In Figure S2, we present qualitative results for object detection, instance segmentation, and semantic segmentation. The results demonstrate that, compared to other methods, our method can achieve more accurate object masks that better align with the ground truth boundaries of the objects.

We would like to express our sincere gratitude for your acknowledgment of our work as both simple and effective, achieving higher performance and efficiency across various frameworks, and providing clear and understandable descriptions of the details of each module in the MEA block. The following is our response with some necessary explanations. If you have any further questions, please feel free to let us know.

Q1: About the number of cascades.

A1: In our work, the number of cascades is set to 16. A detailed description of the specific implementation for the cascaded MEA injector can be found in Section 3.3, and the implementation details of the cascaded MEA extractor are presented in Section 3.4. The details are as follows:

a) In Lines 253-258 of Section 3.3: “Specifically, we first divide the given features into H parts along the channel dimension in a multi-head manner, following the classic self-attention (Vaswani et al., 2017), where H is set to 16 in our work. In the computation process of each head, the output of the h-th head F ̂_sp^(h,i) and F ̂_vit^(h.i) is added into the input features of the next (h+1)-th head F ̂_sp^(h+1,i-1) and F ̂_vit^(h+1,i) to be used in the calculation of subsequent self-attention features, where h = 1, 2, ...,H. The cascade process continues until the feature from the last head is included in the computation.”

b) In line 269 of Section 3.4: “Similarly, the cascaded mechanism is applied following the cascaded MEA injector until the feature from the last head is included in the computation.”

Q2: Different designs of META variants.

A2: For the various sizes of META variants, there are no differences in the implementation details. We maintain the same configurations for the Attention branch, the FFN branch, and the lightweight convolutional branch.

Q3: What specific quantity does MC describe, and what measurements lead to models of different sizes having the same MC?

A3: Sorry for the confusion. MC describes the amount of GPU storage required of the Adapter block to store model parameters, intermediate activations, feature maps, and gradients during both training and inference processes. This metric is closely related to the settings of input image size, batch size, precision, and model architecture. Since we have maintained consistency in these related settings of the Adapter block across different model sizes, this results in models of varying sizes exhibiting the same MC.