QuadEnhancer: Leveraging Quadratic Transformations to Enhance Deep Neural Networks

Thank you for your kind support and valuable suggestions. Some of the additional experiments are still in progress and could not be completed within the rebuttal period. We will update the results and provide clarification in the discussion stage. We apologize for any inconvenience this may cause.

Thank you for your kind support and valuable suggestions. Some of the additional experiments are still in progress and could not be completed within the rebuttal period. We will update the results and provide clarification in the discussion stage. We apologize for any inconvenience this may cause.

W1&Q1: Although it is shown in the introduction that there are other attempts of quadratic based neural network models, including recently, the paper does not offer any experimental comparisons to these. Different architectures may present some challenges here, but this is not discussed. Comparison only include baselines (only ViT and GPT-2 variants), no competitor networks that have quadratic or any additional parametric approaches.

Thank you for your valuale comment. We have added comparisons on image classification with QuadraNets, where the quadratic term is expressed as $y = (x @ W_1)*(x@W_2) + x@W_3$. To ensure fairness, we carefully matched the number of trainable parameters between QuadraNet (QN) and our QuadEnhancer(QE) by adjusting the size of hidden dimensions. The accuracy across several standard benchmarks is summarized below and show that QE significantly outperforms QN in general:

Thank you for your comment again, we have added these comparisons to the revised manuscript.

W2&Q2: The claim in the introduction – on line 60 that QuadraNet V2 has substantial increases computation and parameters does not seem accurate. The QuadraNet V2 paper uses constant training time and shows small increases in number of parameters vs baselines in its experiments. (Maybe it has compute time increase, but the experiments offset this). Please comment. I may have mis-understood this. It raises a slightly broader question of whether this critique is correct of the other previous literature also.

Thank you for your thoughtful observation, and we apologize for the confusion caused by our earlier wording. Our intent was not to suggest that QuadraNets and other work specifically results in prohibitively large training time or parameter counts than linear NNs, but rather to highlight that the parameter and computational complexity of the quadratic components in prior quadratic models is generally higher than that of QuadEnhancer. For example, models in [3,4] typically involve quadratic terms with $O(n^2)$ parameters and computation (some even up to $O(2n^2)$ [1,2,5]), while QuadEnhancer introduces only $O(n)$ parameters and computational overhead for the quadratic term (with $k=1$).

We appreciate you raising this point and have added this clarification in the revised the manuscrip.

W3: There is no comparison in terms of fixed parameter numbers, or fixed training times. (Although the tables show large increases proportional to parameter increases).

Thank you for your insightful comment. We understand the importance of comparisons with fixed parameter counts or fixed training times. To address this:

1. Parameter Comparison: The additional number of parameters introduced by the quadratic models is relatively small compared to their linear counterparts. For example, in the text classification task, we observe the parameter counts are 2.47M vs. 2.45M, 2.83M vs. 2.82M, and 5.40M vs. 5.37M. Given the small difference in parameters, we did not perform a comparison based strictly on identical parameter numbers, as the changes are minimal.
2. Training Time Comparison: To provide further insight into the trade-offs, we have reported training times for two tasks: pretraining on ImageNet-1k and fine-tuning on CIFAR100. The results show that the models with QuadEnhancer (ViT+QE) generally outperform their linear counterparts (ViT) in the later stages of training. Below are the results:

Pretrainign on ImageNet-1k

Finetuning on Cifar100

W4&Q3: The FLOPS comparisons are all theoretical (which are very nice numbers in these comparisons), however, the experimental FLOPS are not stated so we don't get to see if the theoretical model can be implemented in this efficient fashion.

Thank you for your comment. We measured the actual number of FLOPs using NVIDIA's Nsight Compute. We recorded the executed FLOPs for both a standard linear layer and a quadratic-enhanced (QE) layer, each with hidden dimension n=192, averaged over 64 runs. The results are:

This indicates that QE introduces only a ~1.0% increase in actual FLOPs. The theoretical increase is approximately 1/192=0.5%, so the gap is small and reasonable. We note that our current implementation does not yet leverage low-level CUDA or Triton optimization, which could further close this gap.

W5&Q4: Training details only follow a single model from another paper for image classification, and just 20 epochs pre-training, then 10 epochs fine tuning on text classification datasets. This seems a little narrow to understand whether these are sustained results with different numbers of epochs. Are the results robust?

Thank you for your comment. We clarify below:

1. For image classification, the hyper-parameters followed [7]. The authors of [7] have made efforts to tune the hyper-parameters, thus we didn't change their settings. The training runs for hundreds of epochs (300), and the results are stable in the later stages, indicating robustness.

2. For text classification, we used 20 epochs because the model's perplexity had already converged by then. As shown below, further training beyond 16 epochs yields minimal improvement:

And the finetuning of text classifiction is relatively simpler than the pretraining, thus we used 10 epoches, also yielding a similar coverged results at late training stage.

W6: It is slightly surprising that in Tab 8, the 16 rank outperforms 32 in the majority of cases for LLaMA{_,2}-7B. What is the cause of this?

Thank you for your question. The performance drop is due to overfitting. Acctually, this phenomenon also occured in the baseline [6] when the rank was increased to 64. QuadEnhancer, by enhancing the model's capacity, causes overfitting to occur earlier, rather than worsening it.

[1] Xu, Chenhui, et al. "Quadranet: Improving high-order neural interaction efficiency with hardware-aware quadratic neural networks." 2024 29th Asia and South Pacific Design Automation Conference (ASP-DAC). IEEE, 2024.

[2] Xu, Chenhui, et al. "Quadranet v2: Efficient and sustainable training of high-order neural networks with quadratic adaptation." arXiv preprint arXiv:2405.03192 (2024).

[3] Chrysos, Grigorios G., et al. "Deep polynomial neural networks." IEEE transactions on pattern analysis and machine intelligence 44.8 (2021): 4021-4034.

[4] Fan, Feng-Lei, et al. "On expressivity and trainability of quadratic networks." IEEE Transactions on Neural Networks and Learning Systems (2023).

[5] Fan, Feng-Lei, et al. "One neuron saved is one neuron earned: On parametric efficiency of quadratic networks." IEEE Transactions on Pattern Analysis and Machine Intelligence (2025).

[6] Hu, Zhiqiang, et al. "Llm-adapters: An adapter family for parameter-efficient fine-tuning of large language models." arXiv preprint arXiv:2304.01933 (2023).

[7] Mehta, Sachin, Farzad Abdolhosseini, and Mohammad Rastegari. "Cvnets: High performance library for computer vision." Proceedings of the 30th ACM International Conference on Multimedia. 2022.

W1: No mention of gated linear units

Thank you for your insightful comment. We have added clarifications in the revised version, and respond to your concerns in more detail below:

1. Complementary, not competitive.

   • Gated MLP variants like GEGLU and SwiGLU are not direct competitors to QuadEnhancer, and thus are not appropriate baselines for comparison. These gated variants are typically applied within FFNs/MLPs, and involve combinations with activation functions. In contrast, QuadEnhancer is a general mechanism that can be applied to any linear operation across a neural network -- not only FFNs/MLPs, but also convolutions, attention computations, feature projections and other components.
   • Importantly, these approaches are not mutually exclusive—in fact, our experimental setups already include gated modules (i.e., GELUs in ViTs and GPT-2s, and SwiGLUs in LLaMAs).

2. Comparison with a more appropriate baseline

• A more suitable baseline with similar quadratic interaction reformulates linear layers as: $z = (x @ W_1)*(x@W_2) + x@W_3$. We have included new experiments comparing QuadEnhancer (QE) with this baseline (QN) using comparable model sizes. The results, summarized below, show that QE significantly outperforms QN across all datasets in general:

We hope this clarifies the positioning of QuadEnhancer in relation to gated MLPs and highlights its broader applicability and effectiveness.

W2.1: Hard to assess how meaningful the improvement is: The field is full of small changes to MLP layers, and it's usually possible to show improvement by carefully choosing tasks, hyperparameters, or model sizes.

Thank you for raising this important point. We fully understand the concern about the potential influence of selective experimental setups. To address this:

1. Choice of Tasks: We intentionally selected well-established and widely-used benchmarks (though not the latest) in both image classification and language tasks to ensure that our evaluation is grounded and meaningful.
2. Hyperparameters: To maintain fairness, we directly adopted the hyperparameter configurations from prior work [1,2] for the image classification and LLM fintuning tasks, without additional tuning.
3. Model Sizes: Due to limited computational resources, our experiments were conducted with relatively small models. Even though, for LLaMA fine-tuning, the model sizes are still quiet standard and consistent with prior studies.

W2.2 The paper does not provide state-of-the-art results on any benchmark.

Thank you for your thoughtful feedback. We would like to clarify the intended contribution and our perspective on benchmarking:

1. Focus on foundational improvements: Our primary goal is not to achieve SOTA results on specific benchmarks, but rather to propose a more powerful alternative to standard linear operations. We hope this can serve as a useful building block for the broader community in designing more efficient neural networks.
2. SOTA involves many external factors: While SOTA results are certainly valuable, they often depend heavily on task-specific tuning, architectural tweaks, and implementation tricks. We view such results as complementary rather than essential to the core contribution of this work.
3. Future direction: We acknowledge the value of SOTA performance and plan to explore this direction in future work. We appreciate your perspective and will keep it in mind as we continue to develop and evaluate this method.

W3&Q2: No analysis of scaling behavior: Architectural changes often show benefits on small models but lose their impact as model size increases. The experiments in the paper use relatively small-scale models and do not demonstrate behavior on wider range of scales.

Thank you for your thoughtful comment. We have added an experiment that examines the effect of model size by varying the hidden dimension of ViT models trained from scratch on the Caltech dataset:

From the results, we observe two key trends:

• (i) As the hidden dimension increases, both ViT and ViT+QE initially improve in accuracy, followed by a performance drop at the largest size due to overfitting.
• (ii) The gain from QE decreases gradually with model size. This is an expected and reasonable trend, as larger models already possess greater capacity to model complex relationships.

Q1: Am I correct that the ViT architecture used in your experiments has a two-layer MLP with GELU as the nonlinearity, and does not use SwiGLU or GEGLU? If so, why did you choose this configuration, given that GELU-based MLPs are used in older models, and most modern architectures—including recent LLMs—use SwiGLU instead?

Yes, you are correct. ViTs used in our experiments employ GELUs. We chose this configuration to ensure fair comparison by strictly following the original settings from prior work [1], which used GELU-based MLPs. The same principle applies to our LLaMA fine-tuning experiments, where SwiGLU is part of the original architecture and thus remains unchanged. For further discussion on gated MLP variants, please refer to our response to W1.

Q3: Could you clarify how exactly you combined your method with LoRA? If the base architecture remains unchanged (e.g., still uses a two-layer GELU MLP), how does the quadratic enhancement interact with the low-rank LoRA update? Did you modify the A @ B structure to insert a quadratic term between them?

The linear transformation in LoRA-based fine-tuning is expressed as: $y = (W_0 + AB)x$. We treat the combined matrix as a standard weight matrix $W=W_0+AB$, reducing it to a standard linear form $\\tilde{y}=Wx$. This aligns with the formulation in equation (1) of our manuscript, allowing us to apply QuadEnhancer directly on top of the LoRA-adapted weights without altering the LoRA structure itself. The final output after applying QuadEnhancer becomes: $\$ \tilde{y}&=(W_0+AB)x\\z&=(\Lambda \tilde{y}) \odot \tilde{y}+\tilde{y}
$$. Thank you for pointing this out. We have added this clarification in the revised version of the manuscript.

[1] Mehta, Sachin, Farzad Abdolhosseini, and Mohammad Rastegari. "Cvnets: High performance library for computer vision." Proceedings of the 30th ACM International Conference on Multimedia. 2022. [2] Hu, Zhiqiang, et al. "Llm-adapters: An adapter family for parameter-efficient fine-tuning of large language models." arXiv preprint arXiv:2304.01933 (2023).