Unitention: Attend a sample to the dataset

Thank you very much for your strong support and the insightful comments. We appreciate for the comments on "through a novel perspective", "good organization" and "justifies its claim through several good experimental settings". We would like to address your concerns as follows.

I suspect this is not very fair for the testing scenario, as testing should be performed via individual images, one by one. If one intends to include the contextual information from datasets, one also should not include the extra data in the dataset during the testing phase, otherwise, it is not fair. I only have this concern.

A: We thank the reviewers for raising concerns about improving the fairness of our methodology. Despite this concern, we believe that Unitention's test scenario is still fair. The reason for this is as follows:

• 1. It is generally acknowledged that during the testing, the use of training data is fair, whereas the use of test data (e.g., using multiple test samples in an inference, not one by one) is a data leakage. For instance, many machine learning models like k-NN, Decision Tree, SVM, Gaussian Process, are all based on training data's features. At the time of testing, these models already have information about the entire training dataset. They are considered fair, because no features of test data are memorized during the testing.
• 2. When testing, Unitention behaves quite similarly to these traditional machine learning models. It also uses multiple features from the training data (like what k-NN and Decision Tree do) to gain some understanding at the dataset level. So this is also a fair testing.
• 3. We conclude that the reason why Reviewer Ubis has such concerns is Unitention uses a new inference process that is fundamentally different from mainstream deep learning models: it uses a universal-individual feature fusion mechanism, whereas typical deep models have no such explicit dataset-level (universal) features. This reaffirms the originality and uniqueness of our approach.

We have updated our manuscript to explain more on the fairness of Unitention's testing phase. We hope our replies can address your concerns.

 

Note from the authors: we have updated several important experiment results, which we think would be valuable and is also suggested by other reviewers: we now tested Unitention on object detection (MS-COCO 2017), instance segmentation (MS-COCO 2017), and semantic segmentation (ADE20k) tasks. The results are as follows:

The improvements of Unitention are non-trivial among all these dense prediction tasks. These cross-task results, along with the cross-modal results in the paper, provide a comprehensive and robust validation of Unitention. We highlight the results here, and you can also check them out in our updated appendix.

 

---

Last, thanks so much for helping us improve this work through your comprehensive perspective. If you have any additional questions or anything you would like to discuss in more detail, please feel free to let us know. If you find we have successfully addressed your worries, please kindly consider re-evaluating our work, thanks a lot.

Many thanks to Reviewer CKwM's comments and advice, which help us a lot to improve our work. We are also grateful to Reviewer CKwM for evaluating our work as "an interesting exploration", "simple-yet-effective", and perhaps inspiring subsequent works. See below for the answers to your questions and comments.

---

(W1) Although Unitention shares some commonalities with coding algorithm, it is essential to clearly highlight their differences to prevent misleading readers. (1): traditional codebook are unsupervised, while codebook here is dependent on labels

A: We highly appreciate the helpful advice to avoid misleading readers. Traditional codebook motivates our high-level idea of using some universal "bank" to complement the deep feature. This high-level idea of Unitention does not limit the specific bank structure to be implemented. As mentioned in Section 2.2, there are a lot available ways to implement the bank, e.g., label-free FIFO queue, or label-based average class centers. In practice, we choose the class center only for its performance, and this does not contradict our original high-level motivation. We have updated the statements to mitigate any potential misunderstanding.

(W1) (2): unlike the selection of k-nearest neighbor in k-NN, the unitention module instead aggregates all feature candidates ...

A: Thanks for the professional comment. When we design the Unitention module, we don't try to imitate, or copy a k-NN algorithm, but rather to borrow the design philosophy behind k-NN to produce a "universal" understanding of the dataset. As you pointed out, k-NN is performed like a "hard" nearest neighbour selection and is therefore hard to optimize. So we consider using the "soft" version of neighbor aggregation (softmax attention) as a trainable module for our quest for universal understanding. In other words, k-NN builds the universal understanding, but is not the only way to do so.

 

(W2) unitention faces the challenge when migrating to other downstream task with different categories. Additionally, such character should be clearly outlined in section 2.2. Moreover, it is advisable to movie (move) the updating principle for universal banks to section 2.2.

A: Many thanks to the valuable advice. Besides adjusting the according statements in our manuscript, we further clarify about the migration ability of Unitention.

• 1. Unitention is not a pretraining method, and it does not aim to give some pretrained backbones. We tend to think of Unitention as an end application: if someone wants to tackle the classification task on domain B, with a backbone pretrained on domain A, we would suggest to load an existing pretrained backbone (without Unitention), and train it with Unitention directly on domain B (so that the class prototypes are built from scratch on B). This is what we actually do on iNaturalist benchmark, as introduced in Section 3.3 and Appendix D. We will refine Section 3.3 and Appendix D to make this clearer.
• 2. If there is a real need to provide migration capabilities for Unitention, we could use a FIFO queue bank to achieve this. However, this may not perform as well as class centers. This can be explained by the "no free lunch" theorem: generality often comes at the expense of performance.

 

(W3): the training-free approach is like a variation of k-NN. But how does the classifier come from for test phase? By aggregating training samples?

A: We first train a classification model using a standard supervised learning configuration. To obtain a k-NN classifier, we remove the linear classification head of the model, to get a pure backbone $f(\cdot)$.The deep features $f(x\_i)$ of all training samples $x\_i$ are used to build the k-NN classifier.

 

(W4) Lack of some empirical studies. (1): training/inference efficiency

A: We take ResNet-50 training (300 epochs with 2048 batch size) on 8 Tesla V100s as an instance. Unitention adds around 5% extra overhead. Note this can be further optimized by operators such as FlashAttention, as Unitention performs a naive single-query cross-attention:

(to be continued)

(W4) (2): Lack of the evaluation on other popular vision tasks, e.g., detection or segmentation.

A: We highly agree (also stated in the old Limitation section of paper) that evaluation on detection or segmentation tasks can make our empirical conclusion more robust. We have done these experiments and reported the performance in the appendix, where we tested Mask R-CNN ResNet50-FPN on MS-COCO, and the UperNet Swin-T on ADE20k. The results are:

Unitention brings similar gains across these three dense prediction tasks, which showcases its effectiveness.

 

(Q1) It seems that tuning $\tau$ obtains positive gains with momentum 0.8 setting, i.e. row 7, 9, 10 and 11 in Table 4. Why does fix $\tau$ as 1 finally?

A: Thanks for pointing out the typo. In row 9, 10, and 11 in Table 4, $\tau$ should be 1.0. We have corrected this.

(Q2) In Table1, the k-NN accuracy is obtained by the model trained with only k-NN classifier?

A: As explained earlier in the response to (W3), we first train the classification model using standard supervised learning. We then remove the learned linear classification head, and perform k-NN inference on the pure backbone. We have updated the according statements in Section 2.1.

(Q3) It seems that there is a typo in section 3.2, i.e., BL* in tab:imn -> BL* in tab:1.

A: We have fixed this. Many thanks for your thorough proofreading.

 

---

We sincerely appreciate the encouragement and professional, detailed advice from reviewer CKwM. We took all your suggestions into account and updated our manuscript. Please let us know if you have any further questions. If you feel that the above improvements help clear up your doubts and further improve our work, you can kindly consider a re-evaluation, thank you!

(Q1) how does Unitention update the feature bank when the number of class centers is less than number of classes?

A: At the begging of training, we random select 100/1000 classes and only maintain the class centers of them during the whole training. We have updated this detail in ablation study.

(Q2) performance of trainable feature bank?

A: Thanks for such a creative ablation. Following the setting of Section 3.5, we tried it out and found it achieved an accuracy of 79.4%, close to a FIFO queue. We feel that a trainable feature bank would still be hopeless to beat class centers, as class centers directly use the information from labels. This ablation is added to our study and we hope it can help the readers better understand our Unitention.

(Q3) downstream performance of detection and segmentation?

A: See the response to (W2) above.

 

Again, thank you for your insightful and comprehensive advice that greatly improved our Unitention. We have carried out the suggested experiments and made every effort to address the concerns and weaknesses. If there are any further questions or concerns, please let us know. If you feel that we have successfully addressed your concerns, please kindly consider a re-evaluation. Thank you.

---

[r1] Wang, Qilong, et al. Deep cnns meet global covariance pooling: Better representation and generalization. T-PAMI 2020.

[a1] Jiangtao Xie, et al. Sot: Delving deeper into classification head for transformer. arXiv preprint arXiv:2104.10935, 2021.

We thank Reviewer yV5u for their thorough responses and comprehensive suggestions. We appreciate that the effectiveness and simplicity of Unitention is acknowledged. Below we list our responses to each of the comments:

---

(W1) Parameters, flops and inference speeds are not provided.

A: We have reported these in our updated version, hoping they can help the readers better see the efficiency-effectiveness trade-off in our Unitention. For instance, the increases in parameters and flops are:

Overall, we believe the extra overhead is theoretically acceptable. We also measure the wall-clock time and memory footprint of ResNet-50 training (300 epochs with 2048 batch size) on 8 Tesla V100s:

Where Unitention introduces about 5% additional cost. Note this can easily be further reduced by operators such as FlashAttention, as Unitention uses a naive single-query cross-attention.

 

(W2) More experiments besides single-label global-feature-based classification tasks to show the generalization ability.

A: We highly agree to benchmark Unitention on more challenging tasks beyond the single-and-global classification, e.g., the per-pixel-classification (semantic segmentation). As we also expressed in the Limitation section, we would like to test Unitention on dense prediction tasks including detection and segmentation. Now we've finished these and reported the results in appendix, where we tested a CNN (Mask R-CNN ResNet50-FPN) on MS-COCO, as well as a transformer (UperNet Swin-T) on ADE20k:

The consistent performance gains from Unitention showcase its generalization ability.

 

(W3) the authors could show that Unitention is complementary with other classification head designs, for example, iSQRT-COV [r1]

A: Thanks for this valuable advice. Since Unitention offers a way to improve heads that is orthogonal to existing methods (i.e. through a cross-data individual-universal fusion mechanism), it is worth investigating whether it can complement those existing designs. In addition to iSQRT-COV [r1] on ResNet, we also test another different head design (SoT [a1] on Vision Transformers, suggested by reviewer zCBo) to make the results more convincing. The results are as follows and are also included in the appendix:

From the results one can see that Unitention can further improve the performance of other head designs due to its new universal-individual mechanism.

 

(to be continued)

[To other advice/questions about experiments]

More comparisons with existing works [r1-r9]

A: Unitention shows a cross-data mechanism to improve classification heads, different from existing single-data-based head-enhancing modules like [r3-r7]. We highly agree that it is valuable to verify whether Unitention can complement other methods, and this is also suggested by reviewer yV5u. Here, we apply Unitention on two open-sourced model that also experimented on ImageNet (SoT [r7] and iSQRT-COV [a5]), and evaluate the validation accuracy:

The further performance improvements demonstrate that Unitention can complement existing advanced head designs. We have included these results in appendix as extra empirical assessments.

 

How about the extra training cost of the proposed method against the baseline?

A: Take ResNet-50 training (300 epochs with 2048 batch size) on 8 Tesla V100s as an example. Unitention introduces about 5% additional overhead (note there is still room for efficiency improvement, e.g., by using FlashAttention, as Unitention performs a naive single-query cross-attention):

 

I would like to the transferring ability of Unitention to downstream tasks as they are more important in real applications.

A: We appreciate the suggestion. In the original Limitation section, we left the application of Unitention to object detection, instance segmentation, and semantic segmentation as future work. We have now completed these three important downstream evaluations. For the sake of diversity, we test two different families of models (CNNs and Transformers). Specifically, the Mask R-CNN ResNet50-FPN on MS-COCO, and the UperNet Swin-T on ADE20k:

The consistent performance gains show that Unitention can boost the performance of downstream dense prediction tasks, further verifying its effectiveness. These results are attached to appendix too.

 

Thanks again for the valuable comments which makes our Related Work more comprehensive and our experiments more solid and robust. We've elaborated on the suggested experiments, and tried every effort to address your concerns on our methodology and experiments. We hope you could kindly consider a re-evaluation if you find our responses are effective. Thank you.

---

[r1] Mircea Cimpoi, et al. Deep Filter Banks for Texture Recognition and Segmentation. CVPR 2015.

[r2] Relja Arandjelovic, et al. NetVLAD: CNN architecture for weakly supervised place recognition. CVPR 2016.

[r3] Yang Gao, et al. Compact Bilinear Pooling. CVPR 2016.

[r4] Feng Zhu, et al. Learning Spatial Regularization with Image-level Supervisions for Multi-label Image Classification. CVPR 2017.

[r5] Mengran Gou, et al. MoNet: Moments Embedding Network. CVPR 2018.

[r6] Shilong Liu, et al. Query2label: A simple transformer way to multi-label classification. arXiv preprint 2021.

[r7] Jiangtao Xie, et al. Sot: Delving deeper into classification head for transformer. arXiv preprint 2021.

[r8] Ke Zhu, et al. Residual Attention: A Simple but Effective Method for Multi-Label Recognition. CVPR 2021.

[r9] Chao Li, et al. NOAH: A New Head Structure To Improve Deep Neural Networks For Image Classification. ICLR 2023.

---

[a1] Carion, Nicolas, et al. End-to-end object detection with transformers. ECCV 2020.

[a2] Vaswani, et al. Attention is all you need. NIPS 2017.

[a3] Radford, et al. Improving language understanding by generative pre-training. 2018.

[a4] Woo, et al. Cbam: Convolutional block attention module. ECCV 2018.

[a5] Wang, Qilong, et al. Deep cnns meet global covariance pooling: Better representation and generalization. T-PAMI 2020.

We appreciate Reviewer zCBo’s constructive feedback, especially the additional related work which is very helpful in highlighting the uniqueness and novelty of our Unitention. However, the judgement of “the motivation, the idea and the cross-attention design of the proposed Unitention are not new” seems to misunderstand the core difference between Unitention and related work [r1-r9], thus may lead to a misjudgment of the value and novelty of Unitention. We are sorry about this and will clarify this later. We'd like to response to each of concerns as follows.

---

[To additional related literature required]

Thanks for pointing out the work [r1-r9] which helps a lot to consolidate our work. [r1-r2] are Bag-of-Words methods which we have already discussed in Section 4.1/4.3, where we've shown their fundamental difference to Unitention. For head-enhancing work [r3-r9], we did a thorough review and summarized it in a new Related Work subsection in the revised pdf. Now we're addressing the core concern on the novelty of Unitention compared to [r3-r9], as follows.

 

[To concerns on novelty]: “given [r1-r9], the motivation, the idea and the cross-attention design of the proposed Unitention are not new”

(i)  The "not new" is somewhat excessive and could be a misjudgment. First, [r3-r5, r7-r9] didn't use either a cross-attention operator or a cross-data fusion (universal-individual) mechanism.

(ii)  Although [r6] uses cross-attention operator (the same as in DETR [a1] and Seq2Seq Transformers [a2, a3]), it does not use cross-data fusion mechanism at all (which is the core design of Unitention), thus is still fundamental different from Unitention, as explained at the bottom of page 4. We repeat here that 1) the [r6, a1-a3] uses Q, K, V from one single data point and model parameters, whereas 2) Unitention uses key-value pairs from multiple different data points. This difference makes cross-attention operator act as two radically different functions in [r6, a1-a3] vs. in Unitention: one to select different regions of a single image for different recognition targets (one cat's face, two cat's claws, etc.), the other to compare one single data with all other data in the whole training set and make some holistic understanding.

(iii)  A full comparison between Unitention and [existing r3-r9, extra ones we surveyed a1-a5] is summarized as follows:

(iv)  In summary, [r1-r9, a1-a5] can be regarded as additional related work to consolidate our work. They won't devalue the main contribution and novelty of Unitention. Instead, they would reinforce the originality and uniqueness of Unitention, which is evident from the comparison above and also acknowledged by Reviewer CKwM as well as Reviewer Ubis. We have updated our Related Work to discuss and explain this, hopefully clearing up any possible misunderstandings.

 

(to be continued)