Thank you for your detailed and thoughtful review. We appreciate your positive feedback about the significance, methodological soundness, and strength of the empirical evaluation. Below, we hope to address the main concerns you raised in turn.

D1: Interpretability of Learned Relations

Figure 8 in the appendix is intriguing, and interpretability work would probably be an entire new paper. I do want to flag one thing about the claim that "Relational attention in DAT language models encodes human-interpretable semantic relations" from the caption: in many cases, classic transformers, too, encode human-interpretable semantic relations. ... I wonder if there's an example of an attention head which finds some kind of relation that is not normally seen in a classic transformer?

Thank you for your thoughtful comments regarding the interpretability of learned relations. This is indeed an interesting and important question that we’ve begun to explore further. We are glad you found the preliminary exploration in Figure 8 intriguing. As you rightly point out, the relations observed in Figure 8 appear to encode relatively simple semantic similarity between words/tokens, which has also been observed in the attention scores of standard Transformer attention. However, it is important to note that these relations serve a different functional role here: while the attention scores $\alpha_{ij}$ in standard Transformers encode a selection criterion controlling where information is routed, in DAT, the relation vectors $\boldsymbol{r}_{ij}$ serve as the values being transmitted between tokens, controlling what information is routed (with an independent set of attention scores controlling the selection criterion).

We agree with you that finding a single interpretable relational attention head may not be very revealing on its own, and we would be excited to investigate whether relational heads in DAT might encode novel relationships that are not typically seen in classic Transformers. In particular, it would be interesting to understand how these relations form computational circuits that are unique to the DAT architecture---that is, characterize specific computational circuits that use relational heads in unique ways to carry out a specific computation, understanding their functional role on a deeper level beyond just the fact that the relations themselves appear human-interpretable.

For now, this interpretability work is outside the scope of this paper, but we intend to explore it further in future work. One initial step we’ve taken is developing an interactive tool (accessible online) that allows users to load pre-trained DAT models and visualize relational representations at various layers on their own inputs. We plan to include a link to this tool in the final, de-anonymized version of the paper.

---

D2: Question on proof of Representational Capacity Theorem.

The theorem in Appendix A seems correct, although the Debreu representation theorem was new to me ... I'd recommend that the authors provide a paragraph outlining the result at a high level

Thank you for the question and the suggestion to provide further discussion on the Debreu representation theorem. We will add a high-level overview of the Debreu representation theorem and the related literature to improve the clarity and accessibility of the representation result presented in Appendix A

Yes, your interpretation of the result and its proof are correct. The Debreu representation theorem, due to the economics literature, identifies preference relations on a topological (e.g., metric) space with a continuous "utility" function, assuming certain continuity properties of the preference relations with respect to the underlying topology. For us, the key is to extend this idea to a family of query-dependent preference relations in order to specify the attention mechanism. That is, each query is associated with an ordered space, and we require continuity of the family of preference relations with respect to both queries and keys, which we formulate as query-continuity and key-continuity, respectively. From there, the result follows by the approximation properties of inner products of MLPs.

D3: Further discussion of related work

If the authors want to expand a bit, there are two areas where I don't see citations; however, they may not really be essential. One is recent work on how transfomers do seem to represent relations, e.g. "How do Language Models Bind Entities in Context?" (Feng & Steinhardt) and papers that cite it. Another is theoretical ways that have been proposed for representing relations (vector symbolic architecture, etc.)

Thank you for these suggestions. We will incorporate them into our discussion of related work in the final version of the paper.

---

Thank you again for your engagement with our work!

Thank you for your thoughtful review and positive feedback on the overall contribution of our work, especially for highlighting the empirical support for our claims, the contribution to relational reasoning in Transformer architectures, and the importance of structured reasoning in deep learning models. Below, we hope to address the key comments and concerns you raised.

---

C1: Comparison with Baselines

We would like to highlight Appendix C (specifically Section C.1) and Appendix D, where we compare our proposed method to several prior works on relational learning. In particular, we compare our model to PrediNet (Shanahan et al. 2020), CoRelNet (Kerg et al. 2022), and the Abstractor (Altabaa et al. 2024), positioning our work within this ongoing line of research on relational reasoning and extending these prior efforts.

One way to describe the architectures in these prior works is that they incorporate "subtractive" inductive biases that constrain the types of representations the model can compute (see [Ref 24] on the "Relational Bottleneck" for more on this). These strict biases enable strong performance on the benchmarks they were designed for (e.g., the Relational Games benchmark introduced by Shanahan et al. (2020)), but also constrain the models to a narrow domain of applicability. By contrast, our approach in developing the Dual Attention Transformer architecture is "additive", in the sense that it incorporates new explicit relational processing capabilities without constraining existing components of the Transformer architecture, allowing the model to learn to select between the different computational mechanisms available to it based on the task or context, as well as compose them to create flexible and expressive computational circuits.

Despite these differences, we find it valuable to compare DAT against those architectures on controlled synthetic benchmarks to explore the trade-offs of strong inductive biases and evaluate DAT against alternative approaches to relational learning. This was carried out and discussed in Appendix C.1.

Initially, due to space constraints, we deferred this discussion to the appendix. However, with the additional page allowance, we plan to integrate this more detailed comparison and discussion of related works into the main body of the paper. Please also see our response numbered A1 to reviewer oPuj, where we discuss a similar question.

---

C2: Conceptual similarities to message-passing networks

We view the aforementioned line of work on relational learning [Ref. 19, 20, 21, 22, 23] to be the most closely related literature to our work. However, we agree that there are some conceptual similarities between our DAT architecture and message-passing networks, which is a characteristic shared by Transformer models in general. In particular, like standard Transformers, the DAT architecture can be described in the language of the message-passing framework. In message-passing terminology, the messages exchanged in standard Transformer attention encode first-order sensory features of the sender, while in relational attention, the messages encode relational features between the sender and the receiver.

We will additionally incorporate a discussion on the conceptual connections between the DAT architecture and message-passing networks into the paper.

---

Thank you again for your review. We hope we were able to address your main concerns.

Response

Thank you for your detailed and thoughtful review. We appreciate your positive assessment of our work and are encouraged by your recognition of its methodological soundness, strong empirical results, and relevance to the literature on relational inductive biases and Transformer-based architectures. Below, we will aim to respond to the key comments and concerns you raised.

---

B1: Computational efficiency

A potential downside of DAT even within the experiments carried out is the possibility that the additional computations required for RelationalAttention outweigh the gains from parameter efficiency. Though differences are likely marginal, was this something you investigated?

Thank you for raising this important point. Our experiments were designed to carefully control for model size (i.e., parameter count) when comparing the DAT architecture to baselines. In particular, the parameter count is slightly smaller for the DAT model compared to the baselines. While we did not explicitly measure computational cost in terms of FLOPS, we expect the differences to be marginal.

From a practical standpoint, we believe that the more important factor in computational efficiency is the availability of optimized GPU kernels, such as FlashAttention. This gap could perhaps be bridged given interest from the MLSys community to develop optimized kernels for relational attention, but for now, this would be an obstacle to scaling DAT-style architectures in a cost-effective way.

---

B2: Scaling to Larger Models

One potential limitation not fully addressed is whether the performance gains from RelationalAttention would still be significant at very large scales, or if they might be incompatible with optimization tricks for self-attention. However, the consistent improvements across the scales explored are promising and warrant further exploration.

While large-scale demonstrations aren't necessary for this paper, do you have intuitions about how the advantages of DAT might scale?

Our intuition, guided by our scaling results up to ~1B-parameter scales, is that the explicit relational processing capabilities of DAT will continue to provide significant benefits at even larger scales. Our hypothesis is that relational processing is a key computational capability that is useful in many domains and at several levels of abstraction---explicit support for this in the model architecture can enable more efficient learning and greater generalization capabilities.

Of course, confirming this hypothesis requires empirical validation, which we currently do not have the resources to do at our academic institution. As you rightly mentioned, and as discussed above, scaling further introduces new challenges, and the availability of various optimization tricks becomes an important consideration. We hope future work will explore this further.

---

Thank you again for your thoughtful review and your engagement with our work.

Thank you for your thoughtful and constructive review. We appreciate your positive feedback on the originality, significance, and clarity of our work. We are especially grateful for the time and effort you took to thoroughly engage with our work, reading the appendix and making note of typos. We'd also like to thank you for your specific and constructive feedback, which we believe has helped us improve the paper further. Below, we outline the key concerns you raised and attempt to address them.

---

A1: Discussion & Experimental Comparison to Related Work on Relational Architectures (in main body of paper)

Experimental analyses are often performed well but not necessarily contrasted to related methods – this has been deferred to the appendix, but might be better placed in the main paper for visibility

The similarity to [22] is discussed in detail in the appendix, but should be indicated much earlier

We agree that a more detailed discussion of related work in the main text would enhance clarity. Accordingly, we will use the additional page allowance to:

• Integrate the experimental comparison to previous relational architectures [20,21,22], currently in Appendix C, into the main text.
• Expand the discussion of Altabaa et al. [22], currently in Appendix D, to Sections 2.2 and 2.3.

---

A2: Suitability of CIFAR for Evaluating Relational Processing in Vision

CIFAR is very object-centric and a single-object dataset

I’d like to know why the authors think that Image classification on CIFAR benefits from the relational modelling. It clearly does, as we can see in the results

Although CIFAR datasets contain only one object per image, we believe relational modeling is valuable at multiple levels of abstraction, from local patches to object parts and entire objects. The explicit relational mechanisms in our architecture enable the model to process and reason about visual relationships between object parts and image patches. For instance, this allows the model to detect symmetries or represent visual similarities between object parts across different regions of the image.

It can be enough to look at one or very few tokens of a CIFAR image and directly tell what the class would be.

In our models, images are divided into small 4x4-pixel patches (64 tokens per image), making it unlikely that a single token would suffice for classification. Since individual tokens represent very small regions at early layers, the models must consider information from several tokens/patches, including the relationships between tokens. At those early layers, the relations visually compare different patches, which can perhaps be thought of as analogous to applying one patch as a "kernel" or "filter" to another patch in the image. At later layers, tokens may come to represent more global higher-level features, and the relations can represent higher-level relations between object parts. Indeed, the improved results we observe for the DAT architecture demonstrate the utility of the enhanced relational processing capabilities of our architecture, even for the simple CIFAR benchmark.

That said, we agree that more complex tasks (e.g., multi-object detection, tracking, semantic segmentation) would better showcase our architecture’s capabilities. We will add a discussion on this limitation and potential future directions.

Have the authors visualised what relations are modelled? If not, is this possible and could be included? Do they represent any ‘expected’/intuitive relations (e.g. parts of the object)?

Following your suggestion, we visualized the relations learned by the ViDAT model. We find that some relations do appear to represent intuitive "visual similarity" relations between object parts.

To illustrate this, we provide an example of a visualization of the learned relations on an image of a truck at Layer 0 and Layer 4. The patch labeled "source" represents the reference token, and the value annotations indicate sigmoid-normalized relation activations $r_{ij}[\ell]$. The relation activations appear to be high for object parts that are visually similar, especially at earlier layers.

We will add a discussion of these findings in the revised paper.

---

A3: Questions

Are these subspaces particularly chosen? And if yes, how?

The 'feature subspaces' are not predefined; they are learned via $W_{q,\ell}^{rel}, W_{k, \ell}^{rel}$ during training. These are separate from the $W_{q,h}^{attn}, W_{k,h}^{attn}$ weights, which specify the selection criterion of the attention operation.

How many comparisons are performed between two tokens?

This is a hyperparameter of the model, denoted $d_r$. For example, in the 1.3B-parameter language model, $d_r = 128$.

We will revise the main text to make these points clearer.

---

Thank you again for your thoughtful review and your helpful feedback.