Composing Linear Layers from Irreducibles

We thank the reviewer for taking the time to carefully engage with our work, and for the thoughtful suggestions and questions. We answer each question below.

Q1: Pertaining to W1 and with the current implementation, how does the computational speed (both training and inference) of the proposed bivector method compare to the related methods (Dense, BH, LR)?

Currently, rotor layers are slower than dense layers in wall-clock inference runtime due to the overhead of the unoptimized Clifford algebra operations. Inference is about $(\text{width} \times \text{depth})/4$ times slower, where width and depth of our rotor gadget are at most 3 and 2 in our experiments. Training has the extra cost of storing the gradients, but the backpropagation through decomposition (see Figure 3) from applied weights to actual parameters, which involves $\log_2 d$ block-sparse matmuls, where $d$ is the model dimension, can be done on a different GPU while backpropagation proceeds through the rest of the network.

We believe we could not only close this gap, but rotor layers may outperform dense layers, and here is why. The primary goal of our work was to synthesize dense linear layers from exponentially fewer geometric primitives. The real opportunity unlocked by our geometric compotition lies in eliminating memory bandwidth bottlenecks, as also described in our answer to Reviewer tqhq. NVIDIA Research's recent LIMINAL paper (arXiv:2507.14397) identifies three critical insights for LLM inference:

1. Memory capacity requirements are enormous (385GB+ for Llama-405B)
2. Memory bandwidth is the primary bottleneck for user tokens/second
3. DRAM-based architectures offer fundamental efficiency advantages

The reviewer will see that our approach directly addresses finding #2: rather than reading in millions of parameters from memory to GPU (the dominant cost in LLM inference), we can, with enough engineering, generate these weights on-chip by reading only ~1000 geometric parameters and constructing rotors (see Figure 3) on the fly! This trades compute for memory bandwidth—an extremely favorable exchange given that memory, not compute, is the primary constraint as noted in the LIMINAL work and other industry reports.

As an aside, this decomposition reminds us of the Fast Fourier Transform: factoring the DFT matrix didn't immediately reduce runtime, yet this compositional insight later enabled $O(n \log n)$ algorithms compared to $O(n^2)$. Likewise, our construction reveals that dense linear layers have far fewer degrees of freedom than their raw parameter counts suggest, allowing for potential new algorithms to take advantage.

In summary, this direction is highly promising, particularly for scaling up inference efficiency in large models. However, realizing the full benefit would require substantial engineering work such as implementing sparse-aware Clifford libraries and optimizing GPU memory layouts, which is beyond the scope of this paper. For these reasons, we do not compare wall-clock runtime to other baselines.

Q2: With regards to W2, could the authors run a simple experiment, e.g., an MNIST classification with a feed-forward model, using their layer construction method and the baseline/related ones, training the model from scratch (as opposed to using a pre-trained LLM)?

Yes, thank you for this excellent suggestion!

We ran additional expeirments with an MLP architecture on FMNIST, where we replace all dense layers (except the final classification head) with rotor layers and trained both end-to-end from scratch. We performed a hyperparameter search for both and trained each for the same number of epochs. In this setting, the conventional dense layers and rotor layers achieved 89.67% and 88.36% accuracy respectively.

As you noted, full end-to-end training on large models like LLMs is currently infeasible due to limitations in Clifford algebra libraries and their hardware integration. These results on FMNIST suggest that rotor layers can be competitive with dense layers when trained from scratch, giving us encouraging evidence for scalability with more engineering effort.

Suggestion: It would improve the paper if the limitations were stated in the main paper, or at least, if it were mentioned that they are given in the Appendix.

We have now added a reference to this section in the main paper’s conclusion. Thanks for the suggestion!

We thank the reviewer for their constructive comments and thoughtful questions. Below we respond in detail to all questions.

Q1 (line 111): It isn’t clear to me how choosing to consider the subalgebra $\text{Spin}(n)$ still allows you to represent a general, arbitrary linear map.

Thank you for raising this subtle point. As you noted, we use a known result (Lemma $1$ by Hestenes and Sobczyk) which states that a general linear map can be written as a finite sum of the conjugations of multivectors in $\text{Cl}(n)$. This is useful, but directly using multivectors is computationally expensive. To make this practical, we must impose additional structure on the multivectors that can be exploited. The subset that we choose, the Spin group $\text{Spin}(n)$, does not capture arbitrary maps in $\text{GL}(d)$, but rather a subset of maps defined by rotors. In Appendix B.3 (Property 3), we fully characterize the matrices isomorphic to the image of rotor sandwich products as a subgroup of $\text{SO}(2^n)$ induced by the Clifford algebra.

What helps in recovering expressivity is our parallel rotor gadget: using rotor modules acting on different subspaces and aggregating their outputs through pooling, we approximate more general linear maps through superposition. Each rotor contributes an orthogonal transformation on its subspace, and their weighted combination approximates non-orthogonal transformations. While we cannot represent all of $\text{GL}(d)$ exactly (unless we go back to Lemma 1), empirically this is sufficient. We find that this subspace is expressive enough to approximate LLM projection layers.

Q2 (line 256): Do the authors expect that performance would remain comparable if all layers were replaced with clifford algebra layers?

Yes, for smaller models performance does remain comparable to the dense linear layer. We ran additional experiments on FMNIST, where we trained an MLP end-to-end from scratch, replacing all dense layers with rotor layers (except the classification head). We performed a hyperparameter search for both and trained each for the same number of epochs. In this setting, the conventional dense layers and rotor layers achieved 89.67% and 88.36% accuracy respectively.

For large models such as LLMs, full end-to-end training is currently infeasible due to the nascent state of Clifford algebra packages and their limited hardware integration. For this reason, we chose layer-wise replacement, keeping all other weights frozen. When multiple layers are replaced with any approximation method under this scheme, small errors can accumulate across layers.

To our knowledge, there are only two PyTorch-compatible packages that implement Clifford operations. On line 764, we discuss one of the many optimizations we made to the torch_ga package (which we provide in the supplementary material). This allows us to execute key Clifford algebra operations (e.g., wedge product, left contraction, sandwich product) for the larger algebras required in LLMs, but significantly more engineering effort is needed to make full-network training efficient at scale.

Q3 (line 287): Are Clifford layers harder to train or more computationally expensive?

Yes, Clifford (rotor) layers are currently more computationally intensive to train, but more efficient to apply at inference time (with a suitably optimized low-level implementation). For example, on LLaMA-1B and 3B, rotor layers require 35.8M and 82.4M FLOPs (multiplied by a factor $K$ which ranges from 1 to 6 depending on the configuration of rotor gadget), compared to 206.1M and 515.3M FLOPs for standard dense layers. Thus, even in the worst case, it works out favorably.

However, we emphasize that this comparison is not our primary goal. Our core contribution is to synthesize dense linear layers from exponentially fewer geometric primitives, rather than to minimize FLOPS of linear layers directly.

The FLOP reduction is coincidental: the synthesized matrices simply happen to have fewer non-zero entries due to their block-diagonal structure. We were not targeting sparsity; in this specific construction, it is a byproduct of our geometric formulation. It is possible that alternative compositional approaches could synthesize the exact dense matrix with no sparsity (and FLOP) benefits.

The real promise going forward lies in eliminating memory bandwidth bottlenecks. The recent LIMINAL paper (arXiv:2507.14397) from NVIDIA Research identifies three critical findings for LLM inference:

1. Memory capacity requirements are massive (385GB+ for Llama-405B)
2. Memory bandwidth is the primary bottleneck for user tokens/second
3. DRAM-based designs offer fundamental efficiency advantages

The reviewer will see that our approach directly addresses finding #2: instead of loading millions of parameters from memory to GPU (the dominant cost in LLM inference), with the appropriate engineering effort, we can synthesize these weights on-chip from ~1000 geometric parameters! This trades compute for memory bandwidth—an extremely favorable exchange given that memory, not compute, is the primary bottleneck, as noted in the LIMINAL work (and other industry reports).

As an aside, this reminds us of the Fast Fourier Transform: recognizing that the DFT matrix can be factored didn't immediately yield compute benefits, but this compositional structure later enabled $O(n \log n)$ algorithms compared to $O(n^2)$. Similarly, our decomposition reveals that dense linear layers have far fewer degrees of freedom than their raw parameter counts suggest.

To summarize, we match expressivity with exponentially fewer parameters, which coincidentally achieves modest FLOP reduction, and more importantly, opens a path to eliminate memory bandwidth constraints (on the practical side) and to exploit the compositional structure (on the model development side).

Statement about limitations: The authors do not acknowledge limitations or potential negative societal impacts of their work.

We discuss limitations in Appendix A, but have now added a reference to this section in the main paper’s conclusion. Thanks for the suggestion.

Thank you for the insightful question and positive feedback! We answer it below.

Q: Can the authors add more on the runtime of their proposed method and does it compare to the baselines?

Currently, rotor layers are slower than dense layers in wall-clock inference runtime due to the overhead of the unoptimized Clifford algebra operations. Inference is about $(\text{width} \times \text{depth})/4$ times slower, where width and depth of our rotor gadget are at most 3 and 2 in our experiments with minimal use of the structure of our matrices.

We believe we could not only close this gap, but rotor layers may outperform dense layers, and here is why. The primary goal of our work was to synthesize dense linear layers from exponentially fewer geometric primitives. The real opportunity unlocked by our geometric compotition lies in eliminating memory bandwidth bottlenecks, as also described in our answer to Reviewer tqhq. NVIDIA Research's recent LIMINAL paper (arXiv:2507.14397) identifies three critical insights for LLM inference:

1. Memory capacity requirements are enormous (385GB+ for Llama-405B)
2. Memory bandwidth is the primary bottleneck for user tokens/second
3. DRAM-based architectures offer fundamental efficiency advantages

The reviewer will see that our approach directly addresses finding #2: rather than reading in millions of parameters from memory to GPU (the dominant cost in LLM inference), we can, with enough engineering, generate these weights on-chip by reading only ~1000 geometric parameters and constructing rotors (see Figure 3) on the fly! This trades compute for memory bandwidth—an extremely favorable exchange given that memory, not compute, is the primary constraint as noted in the LIMINAL work and other industry reports.

As an aside, this decomposition reminds us of the Fast Fourier Transform: factoring the DFT matrix didn't immediately reduce runtime, yet this compositional insight later enabled $O(n \log n)$ algorithms compared to $O(n^2)$. Likewise, our construction reveals that dense linear layers have far fewer degrees of freedom than their raw parameter counts suggest, allowing for potential new algorithms to take advantage.

In summary, this direction is highly promising, particularly for scaling up inference efficiency in large models. However, realizing the full benefit would require substantial engineering work such as implementing sparse-aware Clifford libraries and optimizing GPU memory layouts, which is beyond the scope of this paper. For these reasons, we do not compare wall-clock runtime to other baselines.

We thank the reviewer for the insightful questions and positive feedback. Below we respond in detail to all questions.

Q1: In theorem 2 and in the LLM layer experiments you have a number of parameters going down by a huge factor. This means that the models become highly compressed by this approach. Does it mean that the learning rate on the compressed models become comparatively slower as the parameters get more entangled? Or else, how do the learning rates compare?

Thank you for the interesting question. Full end-to-end training on LLMs is currently infeasible due to limitations in Clifford algebra packages and hardware integration. To study this question more carefully, we ran additional experiments with a MLP trained on FMNIST, where we replace all dense layers (except the final classification head) with rotor layers. The accuracy as epochs increase are shown below:

Accuracy (%)

From these experiments, we observed:

1. Accuracy improves more quickly during training with dense layers than with rotor layers
2. Both methods ultimately reach a final accuracy of 88-89%

Your intuition is correct. In this setting, the learning rate on the compressed model is comparatively slower than the dense approach.

Q2: How do you explain the fact that your rotor based composition works well with 1 or 2 layers (depth) in experiments but then stops improving (as mentioned in line 318 and following ones)?

Thank you for pointing this out! We realize that Figure 5 can be made clearer. Our current setup can be viewed as fitting to a fixed target functionality of the original dense matrix, where we train only to replace that layer while keeping all other weights frozen. This is different from end-to-end training, where more abstract representations across layers co-adapt. Therefore, the best possible perplexity we can reach is that of the original LLM.

From Table 1, we see this lower bound is 9.85. Looking at Figure 5, we see that after a depth of 2, the PPL is already 10.4--somewhat close to the limit, with a depth of 5 reaching 10.1. The PPL doesn't improve as much beyond depth 2 or 3 because the model is already nearing the lower bound. For clarity, we have added a horizontal line at 9.85 in Figure 5 to emphasize this convergence.

This behavior can be understood more precisely as follows. Since we use pre-trained LLMs, the attention projection matrices (i.e., query, key, and value) may already lie close to the subspace representable by our rotor construction. As we are approximating the functionality of fixed matrices rather than training from scratch, 1-2 rotor layers may capture most of the structure needed to match the pre-existing functionality. Additional layers then provide only marginal refinement, rather than letting us access new transformations.

Q3: Do you think that the composition of primitives as given by your result can be interpretable in some setting (maybe some LLM case) or in some toy example? You say in line 349-350 that "Our goal is to understand the compositional structure of the linear transformation itself" so can you please state in more detail what do we understand now with your method precisely?

Thank you for asking this important question.

First, we now understand that the function space learned by linear layers in LLMs can be represented using objects with $O(\log^2 d)$ geometric degrees of freedom, rather than $O(d^2)$ arbitrary parameters. This suggests that the linear transformations (via dense layers) live near highly structured objects that preserve similar invariances in the data. This gives us a clearer perspective on their intrinsic geometric simplicity.

Second, each bivector-rotor pair in our decomposition operates independently on a $2^n$-dimensional subspace. We now understand that the seemingly monolithic $d\times d$ transformation factors into smaller, independent modules--a structure that is very different from low-rank or block-diagonal approximations. This modularity is discovered rather than imposed. As a result, these transformations can be generated from compact geometric primitives instead of being stored as large matrices.

Taken together, this understanding suggests that the underlying computations in LLMs may exhibit hidden geometric simplicity, and that the models could be implicity discovering and exploiting such structure in the data. While we do not yet claim semantic interpretability of the geometric primitives in large models, this can open promising directions for both interpretability research and the design of more efficient architectures. For an example of how this compositional structure can be exploited for efficieny gains, please see our response to Q3 for Reviewer tqhq.

Q4: Do you know, besides drop in parameters, what is a drop in Kolmogorov Complexity incurred during the learning dynamics by your method compared to others? I'd be curious to know how much information is lost due to gradient descent not being able to fit the models, and I'm wondering if there is a metric or a toy case in which you can give more insights on that, besides computing the accuracy or PPL as you do.

We much appreciate this suggestion to explore the information-theoretic implications of our construction. Our background in this topic is somewhat limited, and certainly not at a level to provide a fully satisfying answer.

From our understanding, Kolmogorov Complexity is upper bounded by the number of parameters. For our general rotor gadget (see Appendix B.2), this corresponds to $2c_1c_2{\log_2 d \choose 2}$, where $c_1$ and $c_2$ are hyperparameters and $d$ is the hidden dimension. For the models used in our experiments, we provide parameter counts of the dense linear layer, rotor layer, and baselines in Table 6, Appendix D.

We would be grateful for any guidance or references to theoretical or empirical results that could help us explore this direction more rigorously. If we are able to formalize a meaningful result, we will include it in the final version and credit the reviewer for the suggestion.