Josh Alman

Poly-attention: a general scheme for higher-order self-attention

Fast RoPE Attention: Combining the Polynomial Method and Fast Fourier Transform

Only Large Weights (And Not Skip Connections) Can Prevent the Perils of Rank Collapse

Two Heads are Better than One: Simulating Large Transformers with Small Ones

Fundamental Limitations on Subquadratic Alternatives to Transformers

Fast RoPE Attention: Combining the Polynomial Method and Fast Fourier Transform