Attention-Only Transformers and Implementing MLPs with Attention Heads

We show that MLP neurons can be implemented by masked, rank-1 attention heads, allowing one to convert an MLP-and-attention transformer into an attention-only transformer.