Indirect Attention: Turning Context Misalignment into a Feature

We appreciate that reviewer 4RHD finds the paper clear and well-written, the theory solid, and results promising. We thank the reviewer for the careful review and thoughtful comments and feedback. Below, we provide response to each of the raised concerns and questions one by one.

W1: Thanks for mentioning this, we have expanded the comparisons table for OSOD tasks in the revised version. We had missed "Semantic-aligned fusion transformer for one-shot object detection (CVPR, 2022)". We still outperform the mentioned work by average of 5% on Pascal VOC unseen, COCO seen and unseen classes, but we underperform it only in Pascal VOC seen classes by 2%.

W2 and W3: As both weaknesses are related to relative positional matrix ($P$) and attention bias function($f$), let us address both together. In our experiments, we initialize $P^0_{ij}$ as the relative position $j - i$. This choice provides a simple inductive prior that encodes symmetric distance information.

Regarding $f$, in our initial experiments, we considered a simpler form for the attention bias $f(P_{ij})$, where it was an identity function that merely broadcasted the raw relative position $P_{ij}$ over attention heads. Compared to this "identity function" approach, employing a lightweight 2-layer MLP for $f(P_{ij})$ significantly improved performance. Regarding the computational cost, let us provide extensive details in Q3 as it is relevant.

Q1. explainability of the correlationship between attention bias and semantic information:

Initially, $f(Pij​)$ operates on a purely structural, relative positional offset $(P_{ij}​=j−i)$. In early layers, its learning is primarily driven by capturing common positional patterns that lead to relevant information. We agree that in the deeper layers, the interpretability becomes more complex (as with nearly all deep learning approaches), as it's no longer just about raw positions but about how the model learns to relate semantically enriched positional features to attention scores. Using an MLP for $f$ indeed introduces additional parameters. However, if it were just about additional parameters, the Double cross-attention model has more layers and more parameters than the indirect-attention-based model (we provide a comparison of the number of parameters in response to question 3).

Q2: Neglection of $W_v$

Thank you for mentioning this, we had forgotten to mention the assumtion of orthogonal at initialization for $W_v$, we have rectified it. The $W_v$ is neglected in these derivations under the common assumption that it is approximately orthogonal at initialization for convenience and tractability, which holds in practice with some initialization methods. This allows norm preservation: $||W_v \epsilon_i|| = ||\epsilon_i||$. However, in our experiments, we do not limit ourselves to this initialization method.

Q3: computation complexity

We provide the computation complexity comparison indirect-attention based model and the double cross-attention model for the OSOD task for low and high resolution images. We do not consider the misaligned self-attention as it performs very poorly.

The problem of computational complexity is inherent to attention mechanisms. However, $f$ (just a linear projection) adds minimal complexity compared to the full attention matrix multiplication (one major source of computation complexity in attention mechanisms).

Regarding performance, the COCO dataset already contains images of varying sizes and aspect ratios, so the evaluation result provided in table 1 is general regarding the image resolution.

Thanks to the reviewer's comments and feedback, we have already made rectifications and expanded in both the paper and appendix. We are reviewing the referenced works and incorporating them into the final manuscript. The connection to state space models and linear attention mechanisms presents an interesting direction for future research.

We appreciate that reviewer vSoh finds our work clear and wellwritten, sees the theoretical analysis as comprehensive, and finds the proposed method novel. We thank the reviewer for the detailed review and valuable feedback and comments. We provide response to the raised questions and concerns in below.

W1: Indeed, our theoretical analysis relies on orthogonal initialization of $W_v$ and unit-variance inputs. These choices are not only common for analytical tractability in theoretical deep learning studies but also reflect some common practices in initializing models, where weight matrices are orthogonalized and inputs are layer-normalized to unit variance at each layer or the input data is normalized at the beginning of the model.

W2: Thanks for mentioning this we have expanded on this in our revised version for the final version.

W3: We agree that NLP tasks would complement the experiments, but as the authors’ main specialization has been more towards computer vision, we have tried to focus on CV task. However, since the theoretical analysis does not depend on a specific data type, it should be applicable to NLP tasks as well.

Q1: Regarding the bias function $f$, during our experiments we observed that moving from a fixed function to a shallow MLP leads to notable gains, but deeper networks did not yield too much improvements. A moderate capacity in $f$ is sufficient to capture useful positional priors without overparameterization and increased computation complexity. Regarding the layer-wise update of the positional matrix $P$, similar to normal DETR models we found that updating it dynamically (i.e., $P^{l+1} = g(o^{(l)})$) leads to better alignment and performance. Overall a depth of 4 layers in synthetic tasks and normal depth of 6 layers (standard for DETR models) gives good balance between performance and computational complexity.

Q2: Yes, we provide the benchmark of IA-detr compared to double-cross attention variant in the following. However the standard attention on the OSOD task performs very poorly (as shown in table 1) so we do not consider it in this comparison.

Q2: We have mainly focused on object detection, but since in the segmentation, mainly the detection head is being replaced by a segmentation head, we expect it to not affect the performance on a segmentation task much.

Q3: In fact in current implementation though not multi-scale extracted from backbone, but we already use mixed multi-scale by simple resizing of backbone’s last layer feature output. So the method already handles mixed multi-scale features.

Q4. Our theoretical analysis does not depend on any specific type of data and we assume the keys and values coming from two different distributions which are well aligned with multi-modal data.

Q5: If we understood the question correctly, normally in a one shot object detection task, there is no fine-tuning at all. So once the model is trained on seen classes we have directly tested on unseen classes without any further fine-tuning.

Q6: In figure 5, the position bias is applied in the Indirect Attention module, but because it clutters the figure we have avoided to depict it in detail.

Q7: No, we do not assume the signal and noise share the same variance. We assume unit-variance signals and the noise $\epsilon_i$ are sampled i.i.d. from $\mathcal{N}(0, \sigma^2 I_d)$.

Thank you to the authors for thoroughly addressing all of my concerns. The clarifications and additional insights provided—particularly around the theoretical assumptions, implementation details, and generalization across tasks are helpful. I believe incorporating these points will further strengthen the paper. In my assessment, this work meets the acceptance criteria for NeurIPS, and I will update my rating accordingly.

We appreciate that reviewer oBvC sees the potential importance of our work, and finds the method novel. We also thank the reviewer for the thoughtful comments, feedback, and constructive advise. The concerns raised are important and we provide responses to raised weaknesses and questions as follows.

Weaknesses:

Motivation and natural occurrence of key-value misalignment: Indeed, explicit key-value misalignment has not been studied in prior literature and at first view it seems more like a bug, and unnatural. Our contribution is to explore this and turn it into a feature. While the setup is intentionally non-standard, we believe it opens avenues for architectures that are more robust or expressive. In existing works such misalignment occurs implicitly in the shape that given two sequences of $A$ and $B$, first they are aligned using a block of cross-attentions to get aligned features $C$, and then a second seperate block of cross-attentions between learnt query and $C$ is used to extract something specific from $C$. In addition to the one-shot detection task provided in paper, this scenario appears implicitly in following works [1, 2, 3]. In the one-shot detection task we show that how the proposed indirect-attention can replace the two blocks of cross-attention while achieving better performance with less number of parameters and less computational complexity. Moreover, it should be noted that query and value are not from the same data distribution in the proposed indirect-attention and query is mainly learnable parameters and even not the same length as the value sequence.

Distinction from Cross-Attention: Section 2.3 puts the effort to distinguish our method from standard cross-attention and we accept that the difference is more naunce and difficult to articulate, and to clarify further in simple words: in indirect attention, all three components of queries, keys, and values are drawn from distinct sources with the query being learnable while in cross-attention there are two distinc sources. Moreover, it can be seen in the synthetic tasks the cross attention performs poorly similar to random predictor.

The scope of real-world evaluation and baselines: We chose one-shot object detection (OSOD) as a representative application where three seperate sequences of object queries, reference image, and query image requires a double cross-attention setup. While OSOD is not the only domain where such misalignment is relevant, it offers an easy setting for introduction of the concept and empirical study. We have included all recent baselines in this area, except Semantic-aligned Fusion Transformer (CVPR 2022), which we acknowledge and will incorporate in the final version. Notably, our method still outperforms it on both COCO and PASCAL datasets by large margin.

Questions:

Q1: Thank you for raising this point. In the revised version prepared for the final submission, we have included a simple figure illustrating the differences between indirect attention (IA), cross-attention, and self-attention.

Regarding the scenarios where IA is applicable, it is particularly well-suited for tasks where predictions over a sequence must be conditioned on another sequence. For example, in arbitrary sorting, the model must sort a sequence not according to a memorized alphabetical or numerical order, but based on a custom ordering provided at inference time, such as ordering [b, c, a, d] where 'b' precedes 'a'.

Similarly, in the One-Shot Object Detection (OSOD) setting, the goal is to detect objects in a target image that belong to the same class as a query image. Cross-attention is less effective in such cases because it aligns and fuses two sequences. While cross-attention can be used in OSOD by introducing an additional cross-attention block as presented in the paper. This adds complexity and computational overhead. In contrast, our paper demonstrates that IA, even with a single attention block, can achieve superior performance.

Q2: In fact, our suggested Indirect Attention is specifically designed for the case of misaligned context and can be easily used in standard transformer blocks when facing scenarios explained in Q1.

Q3: We provide the following ablation on the impacts of positional bias $f(P)$ and positional bias matrix updates by semantic information at each layer as two main components of IA:

Q4 Citing 'alignment attention' paper: Thanks for mentioning the “alignment attention” paper, it is relevant to our line of work and we have included and expanded on it in the related work section in our revised version for final version.

[1]: Fs-detr: Few-shot detection transformer with prompting and without re-training (ICCV, 2023)

[2]: BLIP: Bootstrapping language-image pre-training for unified vision-language understanding and generation (ICML, 2022)

[3]: Align before fuse: vsion and language representation learning with momentum distillation (NeurIPS, 2021)

We appreciate that reviewer qjrB finds our theoretical analysis clear and intuitive, our proposed method simple and generalizable, and the results consistent. We thank the reviewer for thoughtful comments and feedback, and interest in the paper. Below we provide responses to the raised questions and concerns.

W1: We agree regarding the specificity of the synthetic tasks, but the complexity of the synthetic tasks is high enough. We could not find any other paper working on arbitrary sorting and retrieval tasks. Usually, similar papers, as cited in our paper, use simple sorting based on alphabetical or numerical natural ordering which can be memorized quickly. The specificity of the tasks is justified by the specific conditions for which indirect attention is designed.

W2: We provide the ablation in response to question 1.

W3: We provide the response in Question 2 as it is similar.

Q1: We have conducted the requested ablation on Pascal VOC dataset and provide the result as follows. The positional bias is $f(P_{ij})$ as the core part of the model.

Q2: The orthogonal initialization and unit variance input assumptions in theoretical analysis are a common assumption used in similar theoretical analysis for convenience and tractability, and is also sometimes used in practice as well, especially the unit variance is implemented in the form of data normalization and layer norm. However, in our experiments, we have not limited ourselves to these constraints, and the results we provide are not based on orthogonal weight initialization.

Q3: In fact, we think the one-shot objection detection task is a natural and practical example of misalignment where the target image is completely different in size, and objects present in the image from the query image. It is similar to an arbitrary sorting task, where a custom ordering has nothing to do with the given input other than providing a rule for sorting. We agree that application of IA on multimodal (text and image) remains an interesting aspect and a future work.

Q4: We provide an ablation in Question 1, but regarding that if the superior performance is from just a simpler model (though it is a positive point), the standard self-attention (reported in table 1) has also similar simplicity with the same number of decoder blocks but performs very poorly.

Q5: Here is the result of an experiment on a simple synthetic classification task with added noise sampled from $\mathcal{N}(0, I_d)$ to the value sequences only, averaged over 500 data points. Since it is not possible to upload or attach images or PDFs of the curve, we are reporting it in a table.