PolaFormer: Polarity-aware Linear Attention for Vision Transformers

Weakness: First point unclear to me is about the definition of negative values (Line 201). As clearly stated in softmax kernel function (Line 16), they operate in the output end of softmax function where it is deemed that all values are not negative. It is not about the input end of softmax where values can be either negative or positive. Under this context, I do not see anywhere negative values are lost or overlooked as kernel based linear attention is dealing with the softmax-ed space where no negative values exist at all.

It is hard to relate Eq (4) with the loss of negative values as discussed before such as Lines 190-194. Why ReLU based mapping will just operate on latter 3 terms and leave out the first term? This seems to be ad-hoc assumption. I would suggest that the authors provide a more detailed explanation or proof of how ReLU-based mappings specifically affect the terms in Eq (4), and why this leads to information loss.

The other learnable parts to replacing static hand-designed values is good bit but hard to be claimed as addressing major challenges in this context.

Thanks for the question. To address your concerns, we will clarify the definitions and provide detailed reasoning for how PolaFormer handles the issues related to negative values and information loss:

Non-Negative Restrictions in Attention Mechanisms. In standard self-attention, the output is defined as $\operatorname{output}=\mathbf{SV}$, where $\mathbf{S}=\operatorname{softmax}(\mathbf{QK}^\top)$. The softmax function ensures that all elements of $\mathbf{S}$ are non-negative, and each output token becomes a probability-weighted sum of $\mathbf{V}$. While there are no constraints on the signs of individual elements in $\mathbf{QK}^\top$, applying the exponential function in softmax naturally converts all values to positive. As noted in prior works ([1], [2]), a critical property of $\mathbf{S}$ is that all its elements must be non-negative.

In linear attention, where $\mathbf{S} = \phi(\mathbf{Q}) \phi(\mathbf{K})^\top$, non-negativity must be enforced explicitly by the kernel function $\phi$. If this condition is not met, the presence of negative elements in $\mathbf{q}_i$ or $\mathbf{k}_j$ can lead to negative values in $\mathbf{q}_i \mathbf{k}_j^\top$, causing instability in training (e.g., exploding gradients).

The Loss in Existing Linear Attention Methods. To meet the non-negativity requirement, existing linear attention methods often map negative values in $\mathbf{q}$ and $\mathbf{k}$ to zero. However, these approaches overlook the large-magnitude negative values in $\mathbf{q}$ or $\mathbf{k}$. During the inner product computation, the ignored dimensions (where $\mathbf{q}^{-}$ and $\mathbf{k}^{-}$ are zero or close to zero) result in inaccurate $\langle \mathbf{q, k} \rangle$ calculations. As shown in the following formula, only the positive dimensions contribute to the computation, while the dimensions involving negative values are ignored. This severely limits the representational power of the model.

$

\begin{aligned} \langle \mathbf{q},\mathbf{k} \rangle &= \langle \mathbf{q}^{+},\mathbf{k}^{+} \rangle + \langle \mathbf{q}^{+},\mathbf{k}^{-} \rangle + \langle \mathbf{q}^{-},\mathbf{k}^{+} \rangle + \langle \mathbf{q}^{-},\mathbf{k}^{-} \rangle\\ &= \langle \mathbf{q}^{+},\mathbf{k}^{+} \rangle + \langle \mathbf{q}^{+},0 \rangle + \langle 0,\mathbf{k}^{+} \rangle + \langle 0,0 \rangle \\ &=\langle \mathbf{q}^{+},\mathbf{k}^{+}\rangle \end{aligned}

$

How PolaFormer Addresses These Challenges. Our algorithm aims to include all four components in the first line of the equation above. To achieve this, we compute these four components separately. Using the ReLU function, we extract $q^{+}$, $q^{-}$, $k^{+}$, and $k^{-}$, ensuring that $\mathbf{S} = \phi(\mathbf{Q})\phi(\mathbf{K})^\top$ contains only non-negative elements while simultaneously preserving the information from all dimensions of the $\mathbf{qk}$ vectors.

[1] Zhen Qin, Weixuan Sun, Hui Deng, Dongxu Li, Yunshen Wei, Baohong Lv, Junjie Yan, Lingpeng Kong, and Yiran Zhong. Cosformer: Rethinking softmax in attention. In The Tenth International Conference on Learning Representations (ICLR). OpenReview.net, 2022.

[2] Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and Fran¸cois Fleuret. Transformers are rnns: Fast autoregressive transformers with linear attention. In International conference on machine learning, pp. 5156–5165. PMLR, 2020.

Weakness: As this work is about scalability of transformer along with the length of input sequences, what is the range of each experiment. In general, the sequences are not that long for vision tasks like image classification, object detection and segmentation. They are not the best suitable test applications. LRA may give longer sequences which is thus better for test. I would suggest the authors to provide specific sequence lengths for each experiment. Additionally, including experiments on tasks with longer sequences would be helpful to better demonstrate the scalability of this approach. Also, the scalability of the proposed model along with the increase of sequence length is not evaluated, which should be a key aspect for this problem. Including an experiment or analysis that explicitly shows how this model's performance and efficiency scale with increasing sequence length, compared to baseline methods, would be important and insightful.

Thank you for your insightful question and suggestion. Our primary focus is on improving the accuracy of query-key interactions in efficient vision transformers, which is crucial for tasks like detection and segmentation that require precise localization. These tasks often rely on token-to-token correlations, making accurate similarity computations critical.

Scalability in Vision Tasks. While scalability with longer sequences or higher resolutions is not the primary objective, we conducted experiments with varying image resolutions to evaluate the scalability of our approach. Specifically, we tested resolutions ranging from $224^2$ to $384^2$ as reported in Lines 535–537 and Table 6 of the paper. To address your suggestion, we have supplemented these experiments with additional results at a resolution of $512^2$. The results demonstrate that our algorithm maintains consistent advantages as the resolution increases. The reason we did not initially explore even higher resolutions was to ensure a fair comparison with existing works. However, the additional experiments on $512^2$ images further validate the robustness of our method for higher-resolution inputs.

Scalability in NLP Tasks. NLP tasks inherently involve longer sequences compared to vision tasks. To evaluate the scalability of our model in such settings, we performed experiments on the Long-Range Arena (LRA) benchmark. In the revised version of the paper, we have explicitly highlighted the sequence lengths used in different LRA sub-tasks to address this concern.

These results highlight the efficiency of our method in terms of both time and memory consumption for longer sequences, reinforcing the advantages of our linear attention mechanism.

Weakness: Even if one wants to have negative values, what is the fundamental challenge with just changing the corresponding component, such as instead of using ReLU, one can simply use other activation functions allowing negative values such as leaky ReLU, Tanh etc. No discussion on this simple baseline choice is provided. This also implies that the motivation of going for more complex design is thus not strong. That being said, for being actionable, it is suggested that the authors compare their approach against baselines using activation functions that allow negative values, such as leaky ReLU or tanh or shift sigmoid with both negative and positive. This would help clarify the advantages of their more complex design over simpler alternatives.

Thank you for raising this point. As stated in Line 190 of our paper, each element of the $\mathbf{S}$ matrix must be strictly non-negative because it directly relies on the dot products of kernel maps. Allowing negative values in the $\mathbf{S}$ matrix violates this requirement, leading to instability during training. Empirically, when negative values were permitted, we observed gradient explosions, causing the training process to fail entirely. To validate this, we conducted experiments on the Long-Range Arena (LRA) benchmark while allowing negative values in $\mathbf{S}$. However, all attempts resulted in task failure, with the loss diverging to NaN.

Weakness: Table 4: while $\alpha$ in Eq (9) is learnable, why in this table it looks like the value is set to 3/5/7? Or I misunderstand? Besides, please show what are the learned value of $\alpha$ for each other experiment.

Thank you for your question. To clarify, $\alpha$ in our learnable power function is a hyperparameter that controls the magnitude range of the outputs, which are regularized by the sigmoid function in the range (0, 1). Specifically, the power function is defined as: $\mathbf{P} = 1 + \alpha \cdot \operatorname{sigmoid}(w_1, \dots, w_d)$ where only the parameters $w_i$ are learnable. The "+1" term ensures that the power function satisfies the first- and second-order derivative requirements specified in Theorem 1. This design ensures both stability and smoothness in the learning process.

In Table 4, we compare model performance using fixed values of $\alpha$ set to 3, 5, and 7, respectively, to illustrate the effect of different magnitude ranges on downstream tasks. We sincerely apologize for the typo and have corrected it in the revised version. Thank you for bringing it to our attention!

Thank you for reviewing our paper! Before addressing your comments point by point, we would like to re-clarify the some key concepts involved this work to provide a comprehensive context for our explanations:

• Linear Attention: Existing linear attention approaches typically replace the softmax operation over the query-key dot product (softmax$(qk^\top)$) with a carefully designed kernel function $\phi$. This substitution allows the computation of qkv to be reordered, achieving linear complexity and improved efficiency.
• Non-Negativity Constraint and Information Loss: To ensure non-negativity (as the output of softmax is inherently non-negative), these kernel functions often rely on mappings like ReLU that sets specific negative dimensions of $\mathbf{q}$ and $\mathbf{k}$ to zero (detailed examples refer to following replies). This introduces significant deviations from the original query-key dot products and results in partial information loss, which we will discuss in greater detail in subsequent responses.
• Our Contributions. The information loss caused by strict non-negativity constraints is commonly overlooked in prior works, leading to higher entropy and reduced discriminative power in the resulting attention mechanism. Our contributions address these challenges through several innovations:
  1. We introduce a more relaxed framework that separates query-key interactions into same-signed and opposite-signed flows, recovering lost information.
  2. We design a learnable matrix to compensate for the non-negativity constraint by effectively mimicking the subtractive operation between flows.
  3. We incorporate a learnable power-scaling mechanism to control positive sequence entropy, maintaining spikiness and precision.
  4. This relaxed framework is particularly beneficial for vision tasks, such as object localization and segmentation, that demand high precision in query-key interactions.

We hope these clarifications provide a solid foundation for understanding our approach. We will delve deeper into the specifics in our responses to your comments. Thank you once again for your constructive feedback, and please feel free to share any follow-up questions—we are more than happy to discuss any aspect of the work further!

Quesstion4:You theoretically show that your method reduces entropy in the attention distribution. Did you measure the entropy empirically in your experiments to confirm this? It would be interesting to see a plot or some data showing how the entropy changes with your method compared to others.

Thank you for your question. To address this question, we compute the entropy of standard self-attention, linear attention and PolaFormer. Additionally, we visualize the distribution of one row of the Attention Score matrix in the Appendix A.5.

Question5: The attention weight visualizations in Figure 1 are helpful. Could you provide more examples, perhaps showing how PolaFormer focuses on relevant parts of the input in different tasks or layers? This could provide more intuitive insight into how your method works.

Thank you for your question. This will help us to further show the characteristics of accurate similarity calculation and low information entropy of our model. We visualize more examples as fig1 in the Appendix A.6

Weakness 1: Introducing a learnable power function and additional parameters like the polarity coefficients $\mathbf{G}_s$ and $\mathbf{G}_o$ could introduce training challenges, such as sensitivity to initialization or convergence issues. The paper doesn't discuss whether they encountered any of these problems or how they addressed them. & Question1: Did introducing the learnable power function and the polarity coefficients $\mathbf{G}_s$ and $\mathbf{G}_o$ affect training stability? For example, did you encounter issues like vanishing or exploding gradients? If so, how did you mitigate them? & Question2: How sensitive is the model's performance to the initialization and learning of the polarity coefficients $\mathbf{G}_s$ and $\mathbf{G}_o$? Did you notice any significant performance drops or instability when varying these parameters?

Training Stability. We appreciate your insightful questions regarding the training stability and sensitivity of the model to the learnable power function and polarity coefficients, $\mathbf{G}_s$ and $\mathbf{G}_o$. Below, we address each aspect in detail: During our experiments, we initialized $\mathbf{G}$ using the Kaiming uniform initialization by default. We did not encounter any issues related to gradient vanishing or exploding, and the training process remained stable throughout. To further evaluate the robustness of the model, we conducted additional experiments to investigate the effect of different initialization strategies on training dynamics and downstream performance.

Sensitivity to Initialization. To assess the impact of $\mathbf{G}$ initialization on downstream tasks, we conducted additional experiments using five distinct initialization methods. These experiments were performed on a text classification (TEXT) task in Long Range Arena (LRA) with a sequence length of 4k, maintaining the same experimental setup as described in Table 4 (Line 449). The initialization strategies tested included Kaiming uniform, zero initialization, normal distribution ($\mathcal{N}(0, 1)$), uniform distribution ($\mathcal{U}(0, 1)$), and constant ones. The results are summarized in the table below:

These results show that while initialization strategies have varying effects on performance, our algorithm maintains its overall advantage compared to baseline methods. Notably, uniform initialization performed slightly better, suggesting it is a strong alternative to Kaiming uniform. However, none of the initialization methods led to significant instability or convergence issues.

Weakness2: Since attention mechanisms are also fundamental in NLP, it would have been interesting to see PolaFormer's performance on language tasks. The paper focuses solely on vision tasks, so we don't know if the benefits carry over to NLP applications. & Question3：Have you tried applying PolaFormer to NLP tasks like machine translation or language modeling? Since attention is crucial in these areas too, it would be interesting to see if your method provides benefits there.

Thank you for your thoughtful comment. To demonstrate the versatility of PolaFormer, we have validated its performance on the Long-Range Arena (LRA) benchmark, which includes NLP tasks such as TEXT, LISTOPS, and RETRIEVAL (Line 432-452). As shown in Table 4, PolaFormer achieves results that are either better or comparable to state-of-the-art methods on these NLP tasks, highlighting its potential for broader applicability.

Our decision to prioritize vision tasks stems from the fact that efficient vision transformer architectures are predominantly benchmarked under vision settings. This ensures a fair comparison and thorough evaluation within the established scope. Nevertheless, we acknowledge that further validation on large-scale NLP tasks would provide a more comprehensive assessment of PolaFormer’s capabilities.

Weakness3: While they do perform an ablation study (as shown in Table 3), it could be more comprehensive. For instance, they could explore how different values of the scaling factor in the learnable power function affect performance, or how sensitive the model is to the choice of convolutional modules used to increase the rank of the attention map.

Thanks. We have already conducted ablation studies on the learnable power parameter $\alpha$ in Table 4 (Lines 449 to 452). Additionally, experiments on the choice of convolution models are presented in Table 3, where we compared deformable convolutions and depth-wise convolutions. The results indicate that DWC is more suitable for our model and downstream tasks, providing better performance in these scenarios.

Weakness2.4: L248 it is unclear how $\mathbf{G}^s$ and $\mathbf{G^o}$ can be trained since they depend on the batch size $N$ or is $N$ assumed to be fixed? & Question: please clarify how G^s and G^o can be trained

Thank you for your question. The matrices $\mathbf{G}^s$ and $\mathbf{G}^o$ are derived as part of the linear transformations applied to the input $\mathbf{X}$, in a manner similar to the generation of the $\mathbf{Q}$, $\mathbf{K}$, and $\mathbf{V}$. Thus, The batch size $N$ of $\mathbf{G}$ matches the batch size of $\mathbf{X}$. Below, we outline the detailed process:

1. Input Transformation:
   The input matrix $\mathbf{X}$ (of shape $N \times L\times D$, where $N$ is the batch size, $L$ is the sequence length and $D$ is the feature dimension) is passed through a linear layer that maps $D$ to $4D$ to produce $\mathbf{Q}$, $\mathbf{K}$, $\mathbf{V}$, and $\mathbf{G}$:

   \begin{aligned} &\mathbf{Q,K,V,G} = \operatorname{Linear}(D,4D)(\mathbf{X})
   \end{aligned}

2. Splitting $\mathbf{G}$:
   The matrix $\mathbf{G}$ (of shape $N \times L \times d$ for each head) is split into two equal parts, $\mathbf{G}^s$ and $\mathbf{G}^o$, each with shape $N \times L \times \frac{d}{2}$:
   \begin{aligned} \text{Concat}([\mathbf{G}^s, \mathbf{G}^o]) = \mathbf{G} \end{aligned}

3. Attention Flows:
   The attention mechanism operates in two flows:

   • Same-signed Flow ($\mathbf{O}^s$): Computed based on $\mathbf{Q}$ and $\mathbf{K}$ with the same sign.
   • Opposite-signed Flow ($\mathbf{O}^o$): Computed based on $\mathbf{Q}$ and $\mathbf{K}$ with opposite signs.

   Each flow produces outputs $\mathbf{O}^s$ and $\mathbf{O}^o$, which match the shapes of $\mathbf{V}^s$ and $\mathbf{V}^o$:
   \begin{aligned} \mathbf{O}^s, \mathbf{O}^o = \text{Attention}(\mathbf{Q}, \mathbf{K}, \mathbf{V}) \end{aligned}

4. Element-wise Fusion with $\mathbf{G}^s$ and $\mathbf{G}^o$:
   The outputs $\mathbf{O}^s$ and $\mathbf{O}^o$ are fused with $\mathbf{G}^s$ and $\mathbf{G}^o$ through a Hadamard product ($\odot$):
   \begin{aligned} \mathbf{Z}^s = \mathbf{G}^s \odot \mathbf{O}^s, \mathbf{Z}^o = \mathbf{G}^o \odot \mathbf{O}^o \end{aligned}

5. Concatenation and Multi-Head Fusion:
   Finally, the results of the two flows ($\mathbf{Z}^s$ and $\mathbf{Z}^o$) are concatenated and combined across all attention heads to produce the final output:

$

\begin{aligned}\mathbf{Z} = \text{Concat}([\mathbf{Z}^s, \mathbf{Z}^o]) \quad \text{(per head)}
\end{aligned}

$

These operations, including the linear transformation and attention computations, are differentiable and fully trainable within the standard backpropagation. Training $\mathbf{G}^s$ and $\mathbf{G}^o$ is therefore achieved jointly with the rest of the network parameters during end-to-end training. We will incorporate this detailed explanation into the supplementary material to provide additional clarity. Thank you for bringing this to our attention.

Weakness 1: The main dawback of the experiments is that there is not experiment on truly high resolution image, where this method would fully benefit from the linear complexity

Thank you for highlighting this important aspect. To ensure a fair comparison with existing efficient vision transformer architectures, we initially trained the proposed PolaFormer model using the ImageNet-1K dataset under standard resolution ($224 \times 224$) and a higher resolution ($384 \times 384$), as reported in Lines 535–537 and Table 5 of the paper. To address your concern, we extended our evaluation to an even higher resolution ($512 \times 512$), testing both the baseline and PolaFormer under these more demanding conditions. The results, summarized in the table below, show that PolaFormer consistently maintains its advantage, demonstrating robust performance even at high resolutions.

Disscussion. Regarding the significance of high-resolution tests, it is worth noting that the key challenge in vision tasks such as detection and segmentation is not solely the resolution itself but the accuracy of query-key interactions. Effective activation of the query vector for corresponding key vectors ensures that the objects of interest can be captured. PolaFormer’s design, which focuses on recovering accurate relationships in dot products and minimizing information loss, aligns with this aim and enhances performance across resolutions.

Additional Validation: Long Sequence Efficiency. To evaluate the scalability of our model in such settings, we performed experiments on the Long-Range Arena (LRA) benchmark. In the revised version of the paper, we have explicitly highlighted the sequence lengths used in different LRA sub-tasks to address this concern.

These results demonstrate PolaFormer’s efficiency and scalability for both high-resolution vision tasks and long-sequence NLP applications. We appreciate your feedback, which allowed us to highlight the robustness and versatility of our approach in more challenging scenarios. The results will be updated correspondingly in the revised manuscript.

Weakness2.1: It would be useful to specify from the intro that the method is applied with full training -- it is unclear until the experiments that this is not fine-tuning and not a drop-in replacement for softmax attention at inference time

Thanks for your constructive suggestion. We acknowledge that the current version of the manuscript could better highlight the training setup of our method. To clarify:

1. Classification Tasks: The proposed PolaFormer model was trained from scratch on the ImageNet-1K dataset, as described in Section 5.1 (Line 355).
2. Detection and Segmentation Tasks: For downstream tasks such as detection and segmentation, PolaFormer was fine-tuned following the pretraining stage, as detailed in Sections 5.2 and 5.3 (Lines 377 and 414).

We appreciate your feedback and will revise the main body in future versions to explicitly outline these details upfront. Additionally, we will include further explanations in the supplementary material to clearly distinguish the training setups.

Weakness2.2: L133: better = more accurate or faster?

Thank you for pointing out the ambiguity in our summary of related work on efficient vision transformers. In this context, "better" specifically refers to accuracy. While many efficient vision transformer architectures successfully reduce computational complexity, their accuracy often falls short of standard transformers. We will revise this sentence to clearly articulate that the trade-off in existing methods is between reduced complexity and maintaining competitive accuracy.

Weakness2.3: L162: relationship between d and D?

Thank you for your question. To clarify, $D$ represents the dimensionality of the input vectors, while $d$ is the dimensionality of each attention head. In multi-head self-attention, the total input dimensionality $D$ is divided equally among $h$ attention heads. Thus, the relationship is $D = h \times d$. We will update the text to explicitly define this relationship for clarity.

Weakness2.5: Eq (7) maybe not necessary to define what entropy is... Also the argument about better accounting for negative correlation is repeated 4-6x in the paper. By shaving these repetitions you could make room to move the proof to the main paper.

Thank you for your question. Each row of $\phi(\mathbf{Q})\phi(\mathbf{K})^\top$ is not normalized, in other words, it's not a probability distribution, and there is no such property as information entropy. Thus, we defined The Positive Sequence Entropy(PSE) in Eq(7) to use a unified information entropy calculation for all sequences without probability normalization.

Weakness2.6:L300 IIUC d' = d since g() does just a pointwise mapping -- please mention this explicitly

Thank you for your carefully reading. We have mentioned "$d' = d$ since $g()$ does just a pointwise mapping" explicitly in the revised draft.

Weakness2.7:Maybe some useful related work on the power kernels : [Tolias et al, Particular Object Retrieval With Integral Max-Pooling of CNN Activations, ICLR'16]

Thank you for your advice. We have incorporated the discussion of this work into the Related Work section in the revised draft.

Weakness2: Some latest baselines are missing in the paper. For instance, authors should consider incorporating the latest work [1] for comparsion. This baseline also proposes a new linear self-attention to achieve both high expressiveness capacity and low computation complexity. The comparison on image classification or object detection would be helpful for a comprehensive assessment of the propsoed method.

Thank you for your constructive feedback. We appreciate your suggestion to incorporate [1]. However, there are significant considerations that make direct comparison with [1] less appropriate in our case:

Why Agent Attention is an unfair comparison. The method proposed in [1] uses a novel agent attention mechanism to achieve a high expressiveness capacity. As discussed in the Related Work section (Line 145), agent attention introduces agent tokens $\mathbf{A}$ that aggregate and broadcast global information effectively. This method leverages the Softmax properties while incorporating elements of linear attention. However,

1. Its complexity is not strictly linear with respect to sequence length. Agent tokens in [1] scale with the sequence length as the model size increases. While they are fewer in number than the queries, their scaling nature deviates from the linear complexity characteristic of our approach. This makes a direct comparison between the two methods less equitable.
2. Furthermore, [1] does not address the softmax-free problem, which is a core focus of linear attention approach. Since [1] retains the Softmax mechanism for key operations, it diverges from the primary goals of linear attention methodologies.

Our comparions with SOTA (2024) works. In our work, we aimed to ensure a fair and meaningful comparison by including baselines that align more closely with the objectives of linear attention. As shown in Table 5, we benchmarked our method against four state-of-the-art works (Mobile Attention (ICML'24) [2], CAS-ViT (Arxiv'24) [3], Vision Mamba (ICML'24) [4], and SBCformer (WACV'24) [5]) from 2024. These models were selected for their relevance and adherence to the principles of efficiency and scalability, providing a robust framework for evaluating the strengths of our approach.

[1] Han, Dongchen, et al. "Agent attention: On the integration of softmax and linear attention." ECCV 2024.

[2] Zhiyu Yao, Jian Wang, Haixu Wu, Jingdong Wang, and Mingsheng Long. Mobile attention: Mobilefriendly linear-attention for vision transformers. In Forty-first International Conference on Machine Learning (ICML), 2024.

[3] Tianfang Zhang, Lei Li, Yang Zhou, Wentao Liu, Chen Qian, and Xiangyang Ji. Cas-vit: Convolutional additive self-attention vision transformers for efficient mobile applications. CoRR, abs/2408.03703, 2024b.

[4] Lianghui Zhu, Bencheng Liao, Qian Zhang, Xinlong Wang, Wenyu Liu, and Xinggang Wang. Vision mamba: Efficient visual representation learning with bidirectional state space model. In Forty-first International Conference on Machine Learning (ICML), 2024.

[5] Xiangyong Lu, Masanori Suganuma, and Takayuki Okatani. Sbcformer: Lightweight network capable of full-size imagenet classification at 1 FPS on single board computers. In IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pp. 1112–1122. IEEE, 2024.

Weakness3: One minor error in line 98. Negative-negative should be Negative-positive.

Thank you for your carefully reading. We have corrected this error in the revised draft.

Weakness1: After reading the paper, I am still not sure how the non-negativity constraint is perserved. Especially when a learnable matrix $\mathbf{G}$ is applied. Could authors provide more explanation on how the learnable matrix $\mathbf{G}$ can perserve the non-negativity constraint?

Thank you for your attention to the polarity-aware coefficients matrix and your thoughtful question.

1. Strict non-negativity constraint and its limitations. In existing works, ensuring the rigorous non-negativity of the dot product between $\mathbf{q}$ and $\mathbf{k}$ often relies on kernel functions such as ReLU or ELU+1 due to the unpredictability of $\mathbf{q}$ and $\mathbf{k}$ during training. Specifically, these kernel functions, regardless of their design, can lead to significant drawbacks. For example, ReLU maps all negative values to zero, resulting in substantial dimensional loss during dot product calculations. This reduces the contribution of certain dimensions to the similarity, leading to diminished spikiness or discriminativeness in the representation. Alternatively, functions like abs distort the relationship between similarity and opposition, making it difficult to distinguish between aligned and opposing directions in the vector space. Such limitations hinder accurate similarity calculations, which are particularly critical for downstream vision tasks requiring precise discriminative features.

2. Relaxed non-negativity constraint. To address these challenges, we adopt a more flexible strategy aimed at minimizing information loss. Instead of enforcing a strict non-negativity constraint, we separate the same-signed and opposite-signed relationships across dimensions, modeling them independently and subsequently concatenating them for joint learning. This approach allows us to relax the non-negativity constraint, enabling the model to preserve more information and improve query-key interaction accuracy.

3. Learnable Conpensation. The learnable matrix $\mathbf{G}$ plays a pivotal role in compensating for the relaxed non-negativity constraint. By capturing and leveraging the inverse relationships between the two flows (same-signed and opposite-signed), $\mathbf{G}$ facilitates better integration of information. Empirically, our experiments and visualizations show that $\mathbf{G}$ successfully learns the opposing relationships during training, thereby somewhat ensuring non-negativeness in a way that aligns with the dot product calculation. This design not only enhances the information flow between the two channels but also significantly improves overall performance in downstream tasks.