Efficient Modulation for Vision Networks

Dear Reviewer 6eCe,

We are grateful for your insightful comments and constructive suggestions. We address your concerns and questions as below:

1. Tentative explanation that describes that the modulation might lose effectiveness in larger networks. Expanding on this and at the same time showing this in larger networks would make the contribution stronger.

Response:

Firstly, we clarify that while modulation is more suitable for efficient networks, but it is still effective in larger networks as well.

As suggested, we developed a larger variant of EfficientMod by increasing its width and adding more blocks. This variant, named EfficientMod-Large, has approximately 25M parameters. We compare it with related baselines:

As evident from the table, EfficientMod-Large, while utilizing fewer parameters and significantly fewer FLOPs, achieves superior performance. This outcome, achieved without extensive network design optimization or a fine-tuned training recipe due to limited time, underscores the scalability and efficiency of EfficientMod in larger networks.

Secondly, we provide additional details on the tentative explanation that why modulation mechanism can implicitly achieve high dimensions.

Considering the modulation formula as $y=x^2$ (for simplicity, excluding the transformation function in two branches), where $x \in R^d$. This results in:

$y = [x_1^2, x_2^2, x_3^2, ..., x_d^2] \in R^d$. Here, $y = [x_1^2, x_2^2, x_3^2, ..., x_d^2]$ represents a new feature space that is linearly independent of the original input $x= [x_1, x_2, x_3, ..., x_d]$.

With a skip connection, we effectively combine the original $d$-dimensional features with the new $d$-dimensional features, doubling the feature dimension. This is distinct from simply widening the network's width. Through the modulation mechanism, each layer potentially doubles the feature dimension, leading to a significant increase in dimensionality when multiple layers are stacked.

Notice that this is a tentative explanation and hence we placed it in the suppelemntary. A more rigorous analysis and proof would be an exciting avenue for future research, although it may extend beyond the scope of this current work.

2. One of the most curious things about the paper is the ablation results in Table 5 ... since other aspects seem to play even larger role than abs. perf. of 1%.

Response:

We apologize for any confusion caused by Table 5 and appreciate your curiosity regarding its results. To clarify, the performance variations observed in the table are attributed to different factors. The reduction in performance in top rows is due to a decrease in parameters and FLOPs. However, the change from multiplication to summation in our model, which does not alter the parameters or FLOPs, results in a significant accuracy drop of 1%.

To provide a clearer understanding, we have revised Table 5 to include FLOPs and parameters, and have explicitly indicated which components were removed or altered:

This updated table elucidates the critical role of the multiplication operation in our method. Changing from multiplication to summation, while keeping the parameters and FLOPs constant, leads to a 1.0% decrease in accuracy, a notable gap for efficient models. In fact, this performance gap is larger than that caused by reducing parameters and FLOPs (see row 2, 4 and row 5). Row 1 and Row 2 means no convlutional layer in the network, hence no spatial interactions, resulting to significant performance drop. The primary objective of this ablation study was to assess the impact of each component in our design individually. Replacing multiplication with summation does transform our network into a ResNet-like architecture, ignoring detailed designs.

We will post rest responses later.

Dear Reviewer fvW3,

Thanks a lot for your insightful feedback and the time you have dedicated to reviewing our work! We address your concerns as follows:

Weakness 1: In Table 2, there are no latency reported for state-of-the-art efficient models.

Response: Latency (for both CPU and GPU) is primarily reported in Table 1, comparing state-of-the-art efficient models. Table 2 focuses on demonstrating the empirical improvements of introducing attention to our method.

However, to address your concern, we have benchmarked all models in Table 2 (incorporating results from Table 1) and present the updated results here:

As illustrated, our method demonstrates both high speed and promising performance. While a single scale of models may not fully represent the performance-latency trade-off across different models, we invite readers to refer to Figure 3 in our submission for a comprehensive efficiency analysis of our method.

Weakness 2: The proposed method seems simple and more analysis and motivations for the design are needed to understand the principal of the design choice.

Response: We appreciate your perspective on the simplicity of our method. We believe that its simplicity, coupled with effectiveness, could be an advantage . The primary motivation for our design is the suitability of the modulation mechanism for efficient networks, given its computational efficiency and dynamic nature, blending the benefits of convolution and attention. This motivation of modulation mechanism enables EfficientMod to be straightforward yet impactful.

In detail, we further elaborate on our motivations and the analysis behind our design choices to clarify our approach:

Principles and Contributions: At the core of our work, we have summarized the modulation mechanism and further developed the EfficientMod block. This design effectively harnesses the strengths of both convolution (i.e., local representation) and attention mechanisms (dynamics).

Customization of the EfficientMod Block: We have tailored the EfficientMod block to enhance efficiency and effectiveness. This customization involved streamlining the modulation design and simplifying the context modeling process. These refinements are also important in achieving best balance between performance and computational efficiency.

Integration with Attention Blocks: In integrating attention blocks, we strategically placed attention blocks only in the last two stages of the network for computational efficiency.

Weakness 3: Important baselines such as ConvNeXt and Swin Transformer are not included in the comparisons.

Response: We excluded ConvNeXt and Swin Transformer because they were not designed for efficient networks. However, based on your suggestion, we have carefully designed efficient versions of these models and benchmarked them as follows:

Our EfficientMod-xxs model consistenly outperforms both ConvNeXt and Swin Transformer in terms of latency and accuracy. For ConvNeXt, we reduced the embedding dimension from 96 to 36. For Swin Transformer, we adjusted the configuration to embed_dim=32, depths=[3, 3, 9, 3], and num_heads=[4, 4, 8, 8], resulting in ConvNeXt @4M and Swin-T @4M with computational complexities comparable to our EfficientMod-xxs. These models were trained using the EfficientMod training recipe for a fair comparison. Swin-Transformer cannot be easily exported to ONNX format, but it empirically infers much slow then ConvNeXt and our method.

In conclusion, we hope that our responses and the additional results address your concerns effectively. We are grateful for the opportunity to improve our work based on your valuable feedback! We are eager to make any additional revisions that might be necessary and are open to further discussion.

Best, Authors of Submission 8518

We greatly appreciate the insightful feedback from the reviewers. As Weakness 1 and Weakness 3 are responsed by expiermental results, we have also conducted additional experiments to strengthen our arguments and address the identified weaknesses comprehensively for Weakness 2.

In previous rebuttal, we discussed about the motivation for our design. Here we experimentally show the detailed analysis for our designs.

1. Efficiency of Slimming Modulation Design (Sec. 3.2). In addressing the concerns regarding our slimming modulation design, we have integrated an additional MLP layer into the EfficientMod block. This modification was aimed at assessing the impact of slimming on both performance and computational efficiency. Remarkably, this resulted in a notable reduction in both GPU and CPU latency, with a negligible impact on accuracy. This underlines the effectiveness of our slimming approach in enhancing model efficiency without compromising accuracy.

2. Efficiency of simplifying Context Modeling (Sec. 3.2). To further validate the efficiency of our approach in simplifying context modeling, we compared our single kernel size (7x7) implementation against multiple convolutional layers with varying kernel sizes, as the implementation of FocalNet. Our findings reinforce the superiority of using a single, optimized kernel size. This strategy not only simplifies the model but also achieves a better accuracy-latency trade-off, demonstrating the practicality and effectiveness of our design choice.

3. Integrating Attention in EfficientMod (Sec. 3.3): The introduction of vanilla attention in the last two stages of EfficientMod aimed to improve global representation. We adjusted the block and channel numbers to ensure the parameter count remained comparable between EfficientMod-s-Conv and EfficientMod-s. The results highlight that EfficientMod-s not only shows improved performance but also reduced latency, thereby validating our approach in integrating attention for enhanced efficiency.

In conclusion, the additional experiments and analyses affirm the distinct contributions and efficacy of each design element in our model. We believe these results comprehensively address the concerns raised and further underscore the novelty and significance of our work.

We thank you for your constructive feedback, which has been instrumental in refining our paper and clarifying our contributions to the field.

Dear Reviewer fvW3,

Thank you for the time and effort you have devoted to reviewing our work. We have carefully considered and responded in detail to the questions and concerns you raised.

As the author-reviewer discussion deadline is drawing near, we would like to know if our responses have effectively addressed your concerns. Your insights are invaluable to us, and we aim to ensure that our paper meets the highest standards based on your feedback.

We would like to again appreciate for your time for helping strengthen the paper with your suggestions.

Looking forward to your feedback.

Dear Reviewer Q55t,

Thank you for your constructive comments, which we have found immensely valuable. Although the MLP fusion is only a small part of EfficientMod, we appreciate your concerns regarding it (weakness 1,2, and 3) and are glad to discuss it in more detail.

Weakness 1: The authors should elucidate MLP fusion not only from efficiency but also representational ability.

Response: We appreciate your thoughtful recommendations. We examined the effectiveness of slimming the modulation architecture by integrating an extra MLP layer with our EfficientMod block and modifying expansion ratios to retain comparable parameters and FLOPs. That is, the EfficientMod is divided into an EfficientMod block (having an expansion ratio of 1) and an MLP block (with an expansion ratio of (r-1)). Here are the results based on EfficientMod-s-Conv:

As shown in the table, the sperating MLP block just have a marginally better representational ability (80.6 vs. 80.5), which is negligible. By fusing MLP with the EfficientMod block, we are able to increase the inference speed on the CPU and GPU a lot.

Weakness 2: It is imperative to present a comparison between (b) and (c) with the same number of parameters, both in terms of performance and efficiency, and under the same training settings. For example, a comparison between (b), only with fused MLP, and (c)

Response: Thanks for your suggestions. Fig.1 (b) is a general concept that describes our summarized modulation mechanism, and the inner implementation of context modeling is not limited. We take FocalNet as an example and compare it with our method. Table 4 presented a comparison between FocalNet@4M and our EfficientMod-xxs. Here, we customized a FocalNet variant at 12.7M parameters and compared it with our EfficientMod-s-Conv for a more detailed comparison. The results are presented here:

FocalNet (an instance for Fig.1 (b)) performs much worse than ours EfficientMod-s-Conv, in terms of both latency and performance. When fusing the MLP block to the focal modulation block, we reduce the latency of FocalNet to 7.9 ms and 44.0 ms on GPU and CPU, respectively. However, we didn't train the model successfully, maybe a detailed modification is required.

Weakness 3: Discuss whether the transformer block can also benefit from the proposed method for efficiency (at least it is feasible to fuse the MLP into one unified block) Response: It is feasiable to Transformer block to fuse the MLP into one unified block. We thank the reviewer for the insightful questions, which has a great impact to Transformer-based architectures, not only in efficient networks, but also in large models.

Indeed, there are some endeavors that tried to fuse MLP block to a unified block for faster inference. For example, ViT-22B [1] applies the Attention and MLP blocks in parallel for faster inference, instead of sequentially as in the standard Transformer. In detail, the Q, K, V, and first layer in the original MLP block are fused into one layer, the attention-out layer and the second layer in the original MLP block are fused. Also, PaLM [2] also considers this strategy to speed up inference, rewriting the standard formulation of Transformer from $y=x + MLP(LN(x+Attention(LN(x))))$ to $y=x + MLP(LN(x) + Attention(LN(x)))$. As indicated, the unified block results in roughly 15% faster training speed at large scales.

Hence, it is feasible to fuse the MLP block to a unified block for faster inference. Our EfficientMod represents an early exploration of this approach in the realm of efficient networks.

[1] Dehghani, Mostafa, et al. "Scaling vision transformers to 22 billion parameters." ICML, 2023.

[2] Brown, Tom, et al. "Language models are few-shot learners." NeurIPS, 2020.

Weakness 4: In Fig. 8 and 9, one of the notations should be EfficientMod

Response: Thank you for pointing out this. We will amend the legend to correctly represent 'EfficientMod' (not 'MobileMod') in our revised manuscript.

Again, we extend our deepest gratitude for your insightful questions. We hope our responses can address your concerns. Please feel free to share any further questions or concerns if any. We are eager to engage in further discussions and happy to refining our work based on your valuable insights.

Dear Reviewer xsDm,

We greatly appreciate your insightful feedback and the time you have dedicated to reviewing our work! Your comments have been instrumental in enhancing the quality of our research. We address your concerns as follows:

Q6: In Table 1, are the GPU and CPU latency of different models measured on the same device?

Response: Yes, all models were benchmarked on the same device to ensure consistency in our measurements.

Additionally, we conducted benchmarks across various GPU types, as detailed in Supplementary Table 10. This approach was taken to maintain uniformity in the testing environment and minimize latency variances across different GPU models.

Q5: In Table 2, why adding Attention even reduced the FLOPs?

Response: The reason is that we replaced some EfficientMod blocks with Attention blocks to keep almost the same number of parameters, for a fair comparison.

This involved adjusting the expansion ratio in Attention blocks and other hyper-parameters. We believe that the number of parameters plays a crucial role in performance, and maintaining a consistent parameter number is essential for a fair comparison.

Q4: In Table 1, why VAN and FocalNet results are not included? They seem to be the most relevant works.

Response: The comparison (EfficientMod, VAN, and FocalNet) is detailed in our Ablation Study Table 4, rather than Table 1. This is because VAN and FocalNet models do not fall under the category of 'efficient models' and lack small variants.

For the study, we developed a FocalNet variant with 4M parameters (FocalNet@4M), selecting the best-performing variant among several tested. Our analysis in Table 4 demonstrates that EfficientMod not only outperforms FocalNet and VAN in terms of efficiency but also exhibits significantly lower latency on both CPU and GPU.

For your convenience，we included Table 4 in our submission here:

We will be posting our responses to the remaining questions shortly. We look forward to receiving further feedback and engaging in more detailed discussions. Thank you once again for your valuable contributions to improving our research.

Thanks for the rebuttal. But I didn't see the part 2 (if there is).

Dear Reviewer xsDm,

Thanks for your patience since we are working for the experimental results for rebuttal Q2. Here is our rebuttal part 2.

Q3: It could be better to expand Fig 1 (c) in details when explaining the method.

Response: Thank you for your kind suggestions, we would revise the Fig 1 (c) and add more details in it, ensuring that our method is communicated more effectively and comprehensibly.

Q2: It is not clear how much efficiency improvement does each design contribute.

Response:

Thank you for prompting further exploration into the efficiency improvements contributed by each design aspect. We have conducted additional experiments to elucidate this:

1. About the efficiency of Sliming Modulation Design as described in Sec. 3.2. We investigated the efficiency of slimming the modulation design by integrating an additional MLP layer with our EfficientMod block, adjusting expansion ratios to maintain similar parameters and FLOPs. The results indicate a notable reduction in latency with minimal impact on accuracy, demonstrating the effectiveness of this slimming approach:

2. About the efficiency of Simplifying Context Modeling as described in Sec. 3.2. We compared our implementation using a single kernel size (7x7) with multiple convolutional layers of varying kernel sizes, following FocalNet's context modeling approach. Our results show that adopting a single constructive kernel size optimizes the accuracy-latency trade-off, as detailed in the following table:

3. Integrating Attention in EfficientMod as described in Sec. 3.3: To enhance global representation, we introduced vanilla attention in the last two stages of EfficientMod. The adjustments of block and channel number ensured comparable parameters between EfficientMod-s-Conv and EfficientMod-s, with the latter showing improved performance and reduced latency:

These findings affirm the distinct contributions of each design element to the model's overall efficiency.

Q1: Limited technical contributions in the work.

Response: We understand your concerns regarding the perceived technical contributions. While certain implementation aspects like layer fusion and reshaping operations are not our main focus, we emphasize that our primary contributions lie in:

Summarization of Modulation Mechanism: We provide a comprehensive overview of the modulation mechanism in Section 3.1, highlighting its unique position in the landscape of efficient network designs.

Efficient Modulation Design: As detailed in Section 3.2, we introduce the EfficientMod block, a novel building block for efficient networks, leveraging both convolutional and attention mechanisms.

Compared with other efficient network designs, we hold the idea that modulation is unique and distinguished. It can take advantage of both convolution (local representation, efficiency) and attention mechanism (dynamics). Here we list a taxonomy of the representative efficient networks in Table 1 based on their main technical contributions:

This table underscores the distinctiveness of our contribution, positioning the modulation mechanism as a key innovation in EfficientMod. Our work focuses on advancing this aspect, offering a significant step forward in the field of efficient network design.

Dear Reviewer xsDm,

Thank you very much for your insightful review. Your feedback has been invaluable in helping us enhance the quality of our work.

In response to your concerns, we have provided detailed additional results, further expanded our analysis, and revised our manuscript.

As the rebuttal deadline is approaching, we would greatly appreciate any further comments or suggestions you might have. Could you please let us know if our responses and revisions effectively address your concerns? We are committed to continuous improvement and eagerly await your guidance to further refine our work.

Thank you once again for your constructive feedback and support.

Best regards,

Authors of Submission 8518

Dear Reviewer xsDm,

We sincerely appreciate your insightful comments on our submission (ID: 8518). As the review period is nearing its conclusion, we are eager to know if our rebuttal has effectively addressed your concerns and questions.

Should you have any further questions, or if there are additional aspects of our work that you would like us to clarify, please do not hesitate to let us know. Your feedback is invaluable to us!

Best regards,

Authors of Submission 8518

Dear Reviewer xsDm,

Thanks for your feedback. We believe that in our paper and rebuttal, we have provided detailed analyses and enough insightful takeaways. To ease your understanding, we summarize as follows:

insightful takeaways:

1. We summarized the modulation mechanism, and we pointed out that the modulation mechanism could be the most important part other than detailed implementations in related work like VAN and FocalNet. The modulation mechanism brings advantages from bot convolution (local receptive field, efficiency, etc) and attention (dynamics).

2. We introduced the EfficientMod block based on the modulation mechanism, experimental results indicated that our EfficientMod block can serve as an essential building block for efficient networks, supported by Table 1 and Table 6.

In details of the EfficientMod design, we discuss two points:

1. "Sliming Modulation Design by fusing MLP block, generating a unified block". By doing so, we can achieve comparable performance and speed up inference a lot, as supported by our rebuttal. Also, Reviewer Q55t is really interesting in this part and increases the score to 8.

2. "Simplifying Context Modeling by keeping the most constructive component in the design." By doing so, we show that we don't need sophisticated designs (which are also not good for efficiency), a block that inherits advantages from the modulation mechanism can achieve promising performance.

analyses in terms of the design principles:

1. We have detailed the analyses of design principles in our rebuttal 2. This is also acknowledged by Reviewer Q55t, who raised the score to 8.

2. We have presented detailed ablation studies in Section 4.2 ablation studies, and in Table 5. Table 5 is also questioned by Reviewer 6eCe; after our rebuttal, Reviewer 6eCe increased his score to 6.

Most importantly, we believe that our main contribution "summarization of modulation mechanism" and "EfficientMod block" should not be overlooked. With modulation mechanism, we can improve the performance by 1.0% on ImageNet, WITHOUT ANY additional overhead, as shown in Table 5 last row.

Thanks a lot for your time. We hope our clarification can help you better understand our work.

Dear Reviewer xsDm,

We greatly appreciate your constructive suggestions and insightful comments on our work. As the rebuttal deadline is drawing near, we are keen to know if our responses have adequately addressed your concerns.

Please feel free to share any further questions or insights you may have. Your feedback is invaluable to us.