MogaNet: Multi-order Gated Aggregation Network

We first express our sincere gratitude for your time and effort in reviewing our submission. We have made related adjustments in our revision and provided a general response. We invite you to go through the general response because they will probably answer most of your questions.

1. Some Hard-to-Follow Statements.

As stated in the first point above, our work is inspired by an ICLR 2022 work [1], which is the theoretical foundation of this work. Due to page limitations, we cannot put the entire theoretical derivation of multi-order interaction in the paper. The issues and theoretical foundations referred to in this work are not unstandardized concepts but rather issues derived through solid theoretical studies. We agree that we did not fully explore this aspect in our original submission. Thus, we have endeavored to condense and integrate the necessary understanding of it in the revision. Please refer to our general response for the clear illustration and concepts of MogaNet. For definitions and derivations regarding the interaction, please refer to [1].

2. Why Do Previous Models Lack Middle-order Interactions?

To let you better grasp the theoretical starting point of our paper, we put aside the lengthy theoretical derivation, and based on the existing related research, I would like to make some top-down elaboration on the understanding of the interaction in MogaNet. The study in [1] empirically found that current deep neural networks commonly suffer from a middle-order deficient representation bottleneck.

To the best of our knowledge, we suggest that the interaction in practice is mainly influenced by three factors: the data distribution, the loss function, and the network architecture design. [2] found that in computer vision tasks, the ideal interaction curve tends to be the strongest in the low-order, and the middle-order is comparable to the high-order one. This is related to the data distribution property of computer vision tasks. [1] not only provided the theoretical foundation but also presented a loss function to solve this issue. [3] affected the interaction curve by dealing with the data distribution.

As for ViT, we believe that the main reason for the lack of middle-order, despite the use of adaptive data-dependent feature extraction, is the absence of a targeted module design. Global self-attention fails to promote middle-order interaction, which is easy to recognize by its definition, properties, and the findings of prior works.

There is no research on affecting the interactions by network architecture design. Therefore, combining the conclusions of [1] and our empirical analysis in this work, we argue that there is an absence of middle-order interaction in the current deep models. MogaNet is the first attempt to address this issue from the novel perspective of network architecture design.

3. How Does the Design of MogaNet Affect Interaction?

Overall, our main idea is to force the network to encode the interactions that it would originally ignore and, at the same time, to design a feature extraction method that covers all three interactions as much as possible based on the current experience in computer vision. Specifically, the GAP operation in FD(.) extracts the most coarse-grained but primary feature (encoded by the inherent interaction), and the subtract operation extracts the subsidiary feature (encoded by the originally ignored interaction) that the network may have overlooked and then finally supplements the feature to the network. This is the main idea of FD(.), which is also applied to our channel-wise CA(.) module.

As for the Multi-DW(.), we aim to divide the channels into different groups and perform different feature extraction operations in parallel (to characterize different interactions as much as possible). After merging the channels, the context information is extracted through a gating operation to enhance the originally missing high-order interaction as much as possible in an adaptive data-dependent way. The CA(.) is the same idea as FD(.) but performs in the channel space.

Reference

[1] Discovering and Explaining the Representation Bottleneck of DNNs. ICLR 2022.

[2] Architecture-Agnostic Masked Image Modeling--From ViT back to CNN. ICML 2023.

[3] Discovering and Explaining the Representation Bottleneck of Graph Neural Networks from Multi-order Interactions. ArXiv 2022.

4. Why Does CA(.) Not Include FD(.) Operation?

We are delighted that you have noticed our FD(.) and CA(.) designs. In fact, the FD(.) and CA(.) are, as you said, built on the same concept and purpose. You may have recognized that they go hand in hand by going through the general response and other responses on novelty. CA(.) is the first channel-wise operation that reduces the channel dimension to 1 and conducts feature complementation to force the network against its implicitly preferred interactions.

5. Wrong Markup in Fig.5.

We appreciate you pointing out this issue. We have modified this in the revision. Many thanks again.

6. Visualization of Multi-order Interpretation.

Thank you for your constructive feedback. We have provided a visualization comparison and visualization ablation study in Fig.5, Fig.8, and Fig. A4 in our manuscript. You can recognize the effect of MogaNet through these visualization results. Intuitively, the encoded extremely low- or high-order interactions focus on local textures or extremely discriminative parts, not localizing precise semantic parts. The gating operation effectively eliminates the disturbing contextual noise and makes the model effectively locate the semantic patterns.

7. Will MogaNet Be Less Robust?

MogaNet would not lose robustness. the texture portrayed by low-order interaction lacks high-level semantic features. Concretely, taking computer vision as an example, the model needs to face real-world complex scenes. Too much low-order interaction would make the model lose its discriminative capacity, reducing its robustness in real-world applications. Thus, MogaNet would not become less robust. Extensive experiments on various downstream tasks have empirically verified this point.

8. How Does the [1] Losses and MogaNet Pitted Against or Collaborated With Each Other?

Special thanks for your extremely valuable suggestions! We conducted the experiments you designated on CIFAR-100 with 224$\times$224 resolutions and 200 epochs training. The results show that simply combining MogaNet with the loss function in [1] (forcing the network to learn the specific interactions) does not yield performance gains, while Swin-T or ConvNeXt-T have little gains using the low-order or the high-order interaction loss. After analysis, we conclude that the loss function constraint of [1] forcing up a certain interaction might exert a negative effect on MogaNet. There might be two reasons. On the one hand, MogaNet expects to complement the interaction components that the network is originally prone to neglect rather than deliberately elevating a certain part. On the other hand, MogaNet has achieved high performance on CIFAR-100 compared to Swin-T and ConvNeXt, which might be too saturated to optimize by the loss functions. Based on these interesting findings, we plan to conduct more in-depth research based on this interesting result in our future work. We really appreciate your constructive suggestions.

9. Contradict Expressions.

We sincerely appreciate you pointing out this issue. We have modified this in the revision.

In conclusion, we sincerely appreciate all your effort and valuable feedback. If you are satisfied with our response and effort, please consider updating your score. If you need any clarification, please feel free to contact us. We are more than pleased to discuss with you and look forward to hearing back from you!

Reference

[1] Discovering and Explaining the Representation Bottleneck of DNNs. ICLR 2022.

Dear Reviewer XNzj,

Thanks for your timely reply! We sincerely appreciate your thorough review of our revision, additional experiment results, and analysis. We are delighted that you have found our rebuttal addressed the majority of your concerns and raised your rating to 6.

• With your thoughtful and constructive suggestions, this paper has clearly improved and even enlightened us for our future work. Many thanks for your time and effort in helping us make our MogaNet better. Your recognition of the “fascinating starting point”, “important for reproduction”, & experiments of our research is greatly appreciated.
• In addition, we have also provided the supplemented experimental results (please refer to “Response to Reviewer Koyy (PART 2/2)” for details if you are interested) on ablation studies of how the internal module design of MogaNet affects the interactions. The results show that the proposed modules and their internal design can effectively improve the expressive middle-order interactions (0.3n~0.5n) as well as the classification performance. We believe these will benefit the community as a kit-start to consider network architecture design from new perspectives.
• We respectfully believe that this work is particularly attuned to the ICLR community, and we hope that MogaNet can be recognized by more researchers in the community. Thus, if there might be any additional opportunities for us to further discuss and improve our manuscript, we would like to discuss this with you and would be happy to provide more information based on your feedback or further questions.

Again, we express our sincere appreciation for your constructive feedback and your time and efforts in reviewing our work. We would eagerly welcome any further guidance at your convenience!

Best regards,

Authors

I appreciate that you have addressed the majority of my concerns with the revised manuscript, explanation, and supplemented experimental results.

• Regarding R1/R2, we could, of course, refer the reader to the pertinent literature; nonetheless, in my opinion, a succinct and accurate analysis is still required for a speedy understanding of the main idea of this work. And it is good to see the revision has taken this further.
• It's encouraging to see the additional relevant experimental findings about the loss as well. This section (comparisons and related qualitative analysis) might motivate readers in the future.

Based on this, I would like to raise my rating to 6.

We sincerely appreciate all your valuable feedback, and we are delighted about your comprehensive assessment of our paper. We have made numerous adjustments to our submission and provide a comprehensive general response. We invite you to review our revision and general response because they will probably answer most of your questions. All revised information in our revision is highlighted for your convenience.

1. Presentation Clarity for Sec.3.

You have expressed concerns about the presentation of Sec.3 in our paper. We acknowledge that there may have been sections in our paper that were not as clear as they could have been. Your feedback heightened our awareness of the importance of clear communication in research. During the revision process, we realized that the lack of logical clarity seriously undermined the expression of what we were contributing. Therefore, we have revised the paper based on your valuable suggestions. Please refer to the revised content in magenta color for more details.

2. Technological Novelty.

Specifically, MogaNet's network architecture can be categorized into FD(.), Multi-DW(.) and CA(.). The first two together form the Spatial Moga(.) block. You have mentioned the following counterparts: VAN [1], FocalNet [2], LocalViT [3], Conv2Former [4], and PVT-V2 [5]. First, please forgive me for reiterating that MogaNet is from a completely unique starting point compared to these works, and we are not solving the same problem. Second, just at the level of network architecture design, we are also quite apart, and MogaNet is still novel.

(a) For the Spatial Operation: the VAN [1] uses large DW convolution kernels in series to perform convolution operations. FocalNet [2] utilizes DW convolution kernels to perform convolution and gating operations in series for hierarchical feature maps. Conv2Former [4] applies a so-called “convolutional modulation” operation for feature extraction. Different from all of them, MogaNet first uses a DWConv, then performs channel-wise splitting operations, and performs different operations (different sizes of DWConvs and identity connections) for different groups of channels in parallel, followed by a double-branch activated gating operation. This is a completely different design from the aforementioned works. More importantly, we also have different starting points. We have also performed extensive ablation studies and empirical analysis, and our results show that our design achieves exceptional performance gains while being closely aligned with our aforementioned main purpose.

(b) For the Channel Operation: Firstly, starting from LocalViT [3] and PVT-V2 [5], most of the ViT or modern ConvNet models utilize “CFFN” (Convolutional FFN, i.e., 1$\times$1 Conv → 3$\times$3 DWConv → 1$\times$1 Conv) as the CMixer. In other words, this is a consensus now for the community and should not be taken as a reason for the “limited novelty” of MogaNet. Secondly, MogaNet's CA(.) is designed to address the aforementioned macro-objectives, and its design concept is the same as that of FD(.). Meanwhile, to the best of our knowledge, CA(.) is the first channel-wise operation that reduces the channel dimension to 1 and conducts feature complementation to force the network to learn the inherently overlooked features. Combined with the outstanding performance of CA(.) and the provided empirical analysis, we believe that it is certainly evident to verify the novelty of CA(.).

Reference

[1] Visual Attention Network, CVMJ 2023.

[2] Focal Modulation Networks, NeurIPS 2022.

[3] LocalViT: Bringing Locality to Vision Transformers, ArXiv 2021.

[4] A Simple Transformer-Style ConvNet for Visual Recognition, ArXiv 2022.

[5] PVT v2: Improved Baselines with Pyramid Vision Transformer, CVMJ 2022.

3. About Performance Gains.

We fully respect your assessment that the performance gains are not significant. However, we would like to emphasize two points in this regard:

(a) Consistent gains across a broad range of tasks.

We have conducted extensive experiments in mainstream image benchmarks and various downstream tasks. The results show that MogaNet achieves the consistent best results. To be clear, many related prior works have not done that many experiments, and they are also accepted by top conferences. In contrast, MogaNet is rather the more empirically testable work with sound experiments.

(b) Comparison with prior SOTA models.

Taking the small CNN models as an example, we rank the SOTA models by chronological order and count their performance gains on ImageNet-1K. Note that we reproduce RepLKNet-T$^\dagger$ according to the layer setting in SLaK. We observe that, with the exception of the earlier "low-hanging fruit" work, all the SOTAs provide a gain of 0.2~0.7% top-1 accuracy over the previous work (ConvNeXt as the baseline), and some of them even suffer from performance degradation. In contrast, MogaNet-S shows a performance gain of 1.3% accuracy over ConvNeXt-T, which exceeds the results of prior works accepted by top conferences. Meanwhile, we also add the self-supervised pre-training result of MogaNet (using A2MIM [5]) in comparison to ConvNeXt.V2 [6] (using CMAE [6]), indicating that MogaNet can further benefit from self-supervised learning.

Thus, we believe that the gains brought by MogaNet are perfectly adequate and not trivial.

4. Latency Results.

Due to the limited space, we have not shown the latency results in the main test. However, we provide results for the throughput in Appendix C.3, and MogaNet demonstrates a competitive throughput performance. It proves that MogaNet can take into account the throughput efficiency, in addition to the parameter efficiency and FLOPs. Please view Table A3 in Appendix C.3 for details.

5. CA Counterpart Comparison.

In the CMixer, compared to popular channel aggregation modules (SE, GLU, SwiGLU [1]) and specially-designed FFN variants (CMT [2] and RepFFN [3]) on ImageNet-1K based on MogaNet-S. CFFN (1$\times$1 Conv → 3$\times$3 DWConv → 1$\times$1 Conv) served as the baseline in PVTv2 [4]. Note that RepFFN utilizes 4 group convolution kernels (gConv) to reduce the parameters and apply a structural re-parameterization to merge the kernels into the FC layer for inference.

Overall, we sincerely appreciate your effort and valuable feedback. If you are satisfied with our response and effort, please consider updating your score. If you need any clarification, please feel free to contact us. We are more than pleased and are looking forward to hearing back from you!

Reference

[1] DINOv2: Learning Robust Visual Features without Supervision. ArXiv, 2023.

[2] CMT: Convolutional neural networks meet vision transformers. CVPR, 2022.

[3] RepMLP: Re-parameterizing Convolutions into Fully-connected Layers for Image Recognition. CVPR, 2022.

[4] PVTv2: Improved Baselines with Pyramid Vision Transformer. CVMJ, 2022.

[5] Architecture-Agnostic Masked Image Modeling -- From ViT back to CNN. ICML, 2023.

[6] ConvNeXt V2: Co-designing and Scaling ConvNets with Masked Autoencoders. CVPR, 2023.

Dear Reviewer tkKT,

We first greatly appreciate your thorough review and all your valuable feedback. Many thanks for your time and effort in helping us make our MogaNet better. With your constructive suggestions, this work has been clearly improved and even enlightened us for our future work.

Herein, we sincerely invite you to view our response and provide us with your comments. If you are satisfied with our response and effort, please consider updating your score. If you need any clarification, please feel free to contact us. We would be happy to provide more information based on your feedback or further questions.

• In our response, we carefully illustrated and answered all your questions with experiments or detailed explanations. As you have requested, we have conducted additional experiments on MogaNet to show the performance gains and more CA counterparts comparisons. In addition, your concerns about the technical novelty of MogaNet are meticulously explained in our response. Please refer to our tables and explanations for details. Meanwhile, we have incorporated all changes into the revised manuscript for your consideration. We hope your concerns have been addressed.
• Besides, we have also provided the supplemented experimental results (please refer to “Response to Reviewer Koyy (PART 2/2)” for details if you are interested) on ablation studies of how the internal module design of MogaNet affects the interactions. The results show that the proposed modules and their internal design can effectively improve the expressive middle-order interactions (0.3n~0.5n) along with the classification performance gains.

We respectfully believe that this work is attuned to the ICLR community, and we hope that our work can be seen by more researchers in the community. Thus, your rating is tremendously valuable to us. If there might be any additional opportunities for us to further improve our manuscript and potentially increase your rating, we would like to discuss this with you and would be happy to provide more information based on your feedback or further questions.

Best regards,

Authors

We first express our sincere gratitude for your time and effort in reviewing our submission. We have made related adjustments to our submission and provided a comprehensive general response to discuss the novelty and contributions of this work. We invite you to go through our revision and general response first.

Upon reflection, we agree that there are opportunities to strengthen the novelty and clarify the contributions of our proposed method. Here are our responses to the key points raised:

1. Theoretical Understanding of Multi-order Interaction for CV.

Due to the limited length of the paper, we cannot give all the theoretical derivations in the main text. We provide part of the theoretical derivation in the supplementary material. For more details, please refer to the ICLR 2022 work [1] and Appendix B.

As stated in the first point above, our work is inspired by [1], which is the theoretical foundation of our work. Due to page limitations, we cannot comprehensively give the entire theoretical derivation of multi-order interaction in the paper. We have endeavored to condense and integrate the necessary understanding of it in the revision and provide part of the theoretical derivation in the supplementary material (or refer to [1]). Herein, to let you better grasp the theoretical starting point of our paper, we put aside the lengthy theoretical derivation, and based on the existing related research, I would like to make some top-down elaboration on the understanding of the interaction in MogaNet.

Detailed theoretical derivation and empirical analysis are carried out in [1]. The results empirically show that low-order interaction has the representation capacity for local texture, which is robust but deficient in high-level semantic information. High-order interaction shows a strong representation ability for global information, which requires complex global feature interactions, but the portion of images in datasets is not that large. Therefore, high-order interaction has been empirically shown to reduce the model's robustness and is vulnerable to attacks. Middle-order has strong robustness while exhibiting better semantic feature extraction capability than the low-order one.

2. What Should An Ideal Interaction Curve Look Like?

To the best of our knowledge, the ideal interaction curve should cover all the low, middle, and high levels of interactions and be comparable to all these three levels. However, the situation is different in real-world practice. Because real-world scenarios are complex and changeable, different encoded interactions are adept at dealing with different scenarios. Based on existing research, we suggest that the interaction curve in real practice is mainly influenced by three factors: the inductive bias of data distribution, the loss function, and the design of network architectures. [2] found that in computer vision tasks, the ideal interaction curve tends to be the strongest in the low-order, and the middle-order is comparable to the high-order one, which is related to the data distribution property of natural images. But in other data modalities like graphs, the ideal interaction curve should be different, e.g., the middle-order interactions tend to be the highest because of the sub-graph structures. [1] not only first proposed a theoretical explanation but also presented a targeted loss function. [3] affected the interaction curve by dealing with the data distribution. MogaNet is the first attempt to address this issue from the perspective of network architecture design that fits the natural interaction distribution adaptively.

Reference

[1] Discovering and Explaining the Representation Bottleneck of DNNs, ICLR 2022.

[2] Discovering and Explaining the Representation Bottleneck of Graph Neural Networks from Multi-order Interactions, Arxiv 2022.

[3] Architecture-Agnostic Masked Image Modeling--From ViT back to CNN, ICML 2023.

3. How Does the Design of MogaNet Affect the Interaction Curve?

Overall, our main idea is to force the network to encode the interactions that it would originally ignore and, simultaneously, to design a feature extraction method that covers all three interactions as much as possible based on the current experience in computer vision. Specifically, the GAP operation in FD(.) extracts the most coarse-grained but primary feature (encoded by the inherent interaction), and the subtract operation extracts the subsidiary feature (encoded by the originally ignored interaction) that the network may have overlooked and then finally supplements the feature to the network. This is the main idea of FD(.), which is also applied to our channel-wise CA(.) module. They can help the network to learn the inherently overlooked features encoded by the originally overlooked interactions. As shown in the following table, we report top-1 accuracy and the multi-order interaction strength for the ablation studies of the proposed modules based on MogaNet-S. The proposed Multi-order DW(.) and FD(.) for SMixer and CA(.) for CMixer can improve the middle-order interactions (0.3n~0.5n) and the classification performance.

As for the internal designs in Multi-order DW(.), we aim to divide the channels into different groups and perform different feature extraction operations in parallel (to characterize different interactions as much as possible). After merging the channels, the context information is extracted through a gating operation to enhance the originally missing high-order interaction as much as possible in an adaptive data-dependent way. We also report top-1 accuracy and the distribution of interaction strengths for the channel ablation of Multi-order DW(.) as follows. We find that low-order DWConv (1:0:0) prefers low-order interactions while combining the middle- and high-order DWConv (0:1:1) learns more middle- and high-order interactions. Therefore, we can find an optimal channel combination in DWConv layers that reconciles multi-order interactions, i.e., using $C_{l}: C_{m}: C_{h} = 1:3:4$.

In conclusion, thanks again for your constructive feedback, and we have considered your valuable comments for the revision and future work. If you are satisfied with our response and effort, please consider updating your score. If you need any clarification, please feel free to contact us. We are more than pleased and looking forward to hearing back from you!

Dear Reviewer Koyy,

Thanks for your timely reply! We are delighted that you have found our rebuttal feedback valuable and maintain your rating of 6. Your recognition of the originality, quality & clarity, and significance of our research is greatly appreciated. We would respectfully inquire if there might be any additional opportunities for us to further improve our manuscript and potentially increase the rating. Once again, thanks for your constructive feedback, and we would eagerly welcome any further guidance at your convenience!

Best regards,

Authors

We express our gratitude for your valuable review and constructive feedback. We have adjusted our revision and provided a general response to some common issues. We invite you to go through the general response first, as they may have answered some of your confusion, and you will have a new understanding of the novelty and contributions of MogaNet.

Upon reflection, we agree that there are opportunities to strengthen the novelty and clarify the contributions of our proposed method. Specifically, MogaNet's network architecture can be categorized into FD(.), Multi-DW(.) and CA(.). The first two together form the Spatial Moga(.) block. Here, we will address your concern about the novelty from the following three aspects:

1. On Novelty

You raise important points regarding the novelty and differentiation of our method from related work. We agree that aggregating multi-scale features is a common theme in modern ConvNets, or the so-called “convolutional modulations”. While related work explores multi-scale aggregation, we believe our MogaNet is quite novel from both its purpose and the design itself.

Our MogaNet work was inspired by an ICLR22 work [1], which revealed for the first time the problem that existing deep neural networks suffer from a representation bottleneck under a hierarchical theoretical analysis (Multi-order Game-theoretic Interaction) and thereby proposed a loss function to solve this problem. This work was recognized by ICLR2022, but no further attempts have been made to solve this problem. Thus, we are inspired to address this issue from the novel perspective of network architecture design (instead of loss function).

Specifically, deep neural networks show a natural proclivity towards multi-order interactions - the most expressive middle-order interactions are not well encoded in the current learning process, and we attempt to address this issue with network architectures that can force the network to learn to encode more expressive interactions that would have otherwise been inherently overlooked. To facilitate understanding, from a computer vision standpoint, you can roughly, but not quite accurately, interpret this as forcing the network to capture features that were discriminative but inherently ignored.

To the best of our knowledge, we are the first to address this issue [1] with a specific network architecture design and are also the first to approach network architecture design from the multi-order interaction [1] perspective. Therefore, it is clear that we have a completely different starting point compared to other related works. Again, briefly speaking, we are making a new attempt to solve the problem that was first raised by the ICLR 2022 work [1] rather than a merely "borrowed" network design without a top-down guideline. As such, we believe that MogaNet is obviously interesting and novel from an academic research point of view. Moreover, we summarize the performance gains of MogaNet and related ConvNets on ImageNet-1K as follows, indicating the effectiveness and importance of our work in the line of improving ConvNets.

Reference

[1] Discovering and Explaining the Representation Bottleneck of DNNs, ICLR 2022.

3. On the Main Improvement in MogaNet.

The reviewer makes a fair point. Our ablation study could have better disambiguated the contributions of multi-order aggregation (Moga) vs. CA. To clarify, the spatial Moga(.) module and channel CA(.) module collaborate as an integration to achieve our goal, where the expressive but originally overlooked interactions encoded in DNNs can be forced to enhance. As shown in the following table, we ablate the proposed modules by adding them one by one to the baseline (similar to Table 1) and calculate the distributions of interaction strengths. Note that 0.3n~0.5n reflects the favorite middle-order interactions. In particular, CA(.) is embedded to allow aggregation to dynamically attend to useful features. CA(.) is designed to address the aforementioned macro-objectives, and its design concept is the same as that of FD(.). As provided in the new ablation study of our revision, the CA(.) can only perform its best coupled with Moga(.). It empirically indicates that the Moga(.) and CA(.) collaborate as a unified design for the same goal but are designed from different feature spaces (spatial-wise and channel-wise, respectively).

In conclusion, we thank you for pushing us to better position our work with respect to the rising landscape of modern ConvNets. We sincerely appreciate all your valuable feedback to help us improve our work. We believe that the issues raised by the reviewers have been addressed in the revision. We hope that you will reconsider our submission with this response and the revision. If you are satisfied with our response and effort, please consider updating your score. If you need any clarification, please feel free to contact us. We are more than pleased and looking forward to hearing back from you!

2. On Discussion of Other Convolutional Modulations.

You have mentioned the following similar works: VAN [1], FocalNet [2], and Conv2Former [3] (an occurrent work as MogaNet in late 2022). While related literature exists, the intention and starting points behind our methods are actually far apart. Firstly, MogaNet is from a completely unique starting point compared to these works, and we are not solving the same problem. We aim to solve this obstacle within DNNs by architecture design. We actually own completely different motivations and targets, so we don’t discuss them in this work as the available pages are limited. Second, just at the level of network architecture design, we are also quite apart, and MogaNet is still novel and important:

(a) For the Spatial Operation: the VAN [1] uses large DW convolution kernels in series to perform convolution operations. FocalNet [2] utilizes DW convolution kernels to perform convolution and gating operations in series for hierarchical feature maps. Conv2Former [3] applies a so-called “convolutional modulation” operation for feature extraction. Different from all of them, MogaNet first uses a DWConv, then performs channel-wise splitting operations, and performs different operations (different sizes of DWConvs and identity connections) for different groups of channels in parallel, followed by a double-branch activated gating operation. This is a completely different design from the aforementioned works. More importantly, we also have different starting points. We have also performed extensive ablation studies and empirical analysis, and our results show that our design achieves exceptional performance gains while being closely aligned with our aforementioned main purpose. In addition, MogaNet's performance is far above them.

(b) For the Channel Operation: First, starting from LocalViT [4] and PVT-V2 [5], most of the ViT or modern ConvNet models utilize CFFN (Convolutional FFN, i.e., 1$\times$1 Conv → 3$\times$3 DWConv → 1$\times$1 Conv) as the channel feature extractor. In other words, this is a consensus now for the community and should not be taken as a reason for the “limited novelty” of MogaNet. Second, MogaNet's CA(.) is designed to address the aforementioned macro-objectives, and its design concept is the same as that of FD(.), which is forcing the network to learn to encode more expressive interactions that would have been inherently (originally) overlooked. Meanwhile, to the best of our knowledge, CA(.) is the first channel-wise operation that reduces the channel dimension to 1 and conducts feature complementation. Combined with the outstanding performance of CA(.) and the provided empirical analysis, we believe that it is certainly evident to demonstrate the novelty of CA(.).

We appreciate your helpful feedback and will add a related description in our revision. You rightly note that we could have better situated our work within the broader context of convolutional modulations. We have revised our Sec.1, Sec.2, and Sec.3 in our revision. As shown in the following table, we conduct interaction analysis experiments, which show that MogaNet not only surpasses the popular modern ConvNets you mentioned but exhibits better interaction performance, where the expressive middle-order interactions (0.3n~0.5n) are effectively enhanced. View Appendix B.1 and Fig. A2 for details.

Reference

[1] Visual Attention Network, CVMJ 2023.

[2] Focal Modulation Networks, NeurIPS 2022.

[3] A Simple Transformer-Style ConvNet for Visual Recognition, ArXiv 2022.

[4] LocalViT: Bringing Locality to Vision Transformers, ArXiv 2021.

[5] PVT v2: Improved Baselines with Pyramid Vision Transformer, CVMJ 2022.

Thanks for your timely and detailed reply! We appreciate your thorough review of all our revised manuscript, explanation, and experimental results.

1. About Convolutional Modulation Network.

We are delighted that you have found that our rebuttal analysis is good. Based on your suggestion, we have cited and discussed the works you mentioned in the related content to our latest revision (Appendix E).

2. The Significant Difference between MogaNet and FocalNet.

First, we leave aside the point that the two share completely different research motivations and novelties, albeit we believe this is truly at the heart of the ICLR community. Here, we just consider the network architecture details of both.

• As for FocalNet, it mainly consists of two parts: (i) Hierarchical contextualization, implemented using a stack of seriously connected depth-wise convolutions, to gradually encode multi-scale long-range contexts. (ii) Gated aggregation, which is also designed in series and is implemented as the vanilla gating operation. Last, all the outputs are aligned to add up to the final output. Its distinctive point is that it is designed only from the classical CV perspective, aiming at capturing the long-range feature relationship under the pure ConvNet architecture. Also, it is crucial to emphasize that the FocalNet work never mentions the word "ORDER" in the whole paper, nor the fact that ORDER has completely different meanings in different contexts and research areas. Thus, we objectively believe the fact is that FocalNet is a fantastic work that has nothing to do with MogaNet.

• As for our MogaNet, it primarily comprises the spatial Moga block and CA module.

  (i) For the spatial Moga block, we first employ the FD module to force the network to learn the inherently overlooked features (different from FocalNet). Next, a DWConv followed a channel-wise splitting operation is designed (different from FocalNet). For different groups of channels, we design different operations comprising different scales of DWConvs and identity connections in parallel (unlike that of FocalNet). Along this line, a double-branch SiLU activated gating (unlike that of FocalNet).

  (ii) For the CA(.), it is designed with the same purpose as the FD module but is performed within the channel space. To the best of our knowledge, CA(.) is the first channel-wise operation that reduces the channel dimension to 1 and conducts feature complementation to force the network to learn the inherently overlooked features. Objectively, this makes CA(.) technically novel and efficient. Combined with CA(.) 's outstanding performance, clear visualization, and empirical analysis, we believe that it is certainly evident to distinguish CA(.) from other counterparts.

3. On the Major Contribution.

We first apologize that our expression has caused you confusion. We would like to address your question from two perspectives.

• The New Table Results. First, as you certainly know, modern ConvNets can be interpreted as mainly comprising two parts: spatial mixing and channel mixing. Our modules, Moga(.) and CA(.), are designed as these two components, respectively. Among them, Moga(.) is composed of FD(.), Multi-order DW(.), and the gating branch. As shown in our new table, in terms of accuracy, Moga(.) or the SMixer brings a performance gain of +0.7, slightly above that of the last row CA(.). Besides, considering the interaction effects, we can see from the 3rd line Multi-order DW(.) 's interaction result, which clearly shows that the middle-order interactions (0.3n~0.5n) are effectively improved by the spatial module, along with the performance gains. Thus, we indeed have not misrepresented the main contribution of MogaNet.

• Re-clarification of the Main Contribution and Concepts of MogaNet.

  Holistically, our main purpose is not to propose a specific network module, but to explore visual network architecture design that can force the network to learn to encode more expressive interactions that would have been inherently overlooked. As we have stated, this issue was first revealed by the ICLR 2022 work, but no research tried to explore network design from this angle. Based on this concept, we design MogaNet to instantiate it on both spatial Moga(.) and channel CA(.) spaces, respectively. Therefore, it is a new kick-start for network design from an angle that has not been explored. We share a different starting point with other works. In short, the macro concept with related instantiations is our main contribution. We have conducted extensive experiments and analysis to empirically prove all the instantiations are effective toward our goals.

We express our gratitude for your time and effort. We believe that the more the truth is debated, the clearer it becomes. Hope your issues have been explained and resolved.

Dear Reviewer 6yMg,

Thanks again for your feedback! We hope you have noticed our latest reply. We would like to emphasize that we highly treasure your valuable responses and suggestions, and found the issues you raised are crucial for us to better scrutinize and improve our work. We have revised our work and answered all your questions. Thus we sincerely invite you to view our response again at the end of the rebuttal period. We hope that after you check this out, you will remove some of the misconceptions about this work. Here, we would like to summarize our response above to give you an overview and make our final statement.

• As you requested, we have added analysis and descriptions of convolutional modulation networks to our final revision (e.g., Appendix E).
• Your concerns about the technical difference between our MogaNet and you have mentioned FocalNet are meticulously explained in our response. We have disassembled all the components of these two models to compare them one by one in our response. We hope your concerns have been addressed.
• We have re-interpreted the implications of the results provided in the new table. You may have incorrectly understood what we were trying to convey before. Our spatial Moga(.) brings a performance gain of +0.7, which is slightly above that of CA(.) for the CMixer. Meanwhile, we have conveyed the main idea of this work in simple terms. View our "Additional Response to Concerns" for details.

At the current version, we have been particularly sincere and diligent in responding to your valuable comments. We trust that our response has clearly answered your questions, and we invite you to review it. We respectfully believe that this work is attuned to the ICLR community, and we hope that our work can be seen by more researchers in the community. Thus, your rating is extremely valuable to us. If there might be any additional opportunities for us to further improve our manuscript and potentially increase your rating, we would like to discuss this with you and would be happy to provide more information based on your feedback or further questions. Thanks again for your efforts!

Best regards,

Authors