Quantized Approximately Orthogonal Recurrent Neural Networks

There is unfortunately not any real innovation in the paper. The only contribution of the paper is applying QAT with a straight-through estimator (a very old idea in quantization literature) to existing optimization methods for learning ORNNs. In fact, the QAT technique is not even state-of-the-art as they could learn the quantization ranges $\alpha$ through gradient by adopting LSQ$^{[1]}$ or PACT^{[2]}.

The theoretical analysis of the impact of quantization in orthogonal matrices is not optimal. It is well known that using MinMax quantization for low-bit quantization (< 6 bits) leads to significant degradation. This is why search-based methods are normally adopted that find the scale or range that minimizes the Forbenious ( or other norms) between quantized and unquantized vectors (commonly known as MSE-based ranges). It would be more interesting to see the plots of Figures 1 & 2 using optimal $\alpha$ values rather than ones based on the maximum range.

[1] LSQ: Learned Step Size Quantization, Esser et al. [2] PACT: Parameterized Clipping Activation for Quantized Neural Networks

• The proposed methods seem very straightforward: combine the strategy of constructing ORNNs with STE, which is a standard and well-known technique in the quantization literature.
• L286-L300: authors derive bounds for approximate orthogonality of q(W) which they themselves state are too loose to be useful in practice. It would be quite insightful to track and report $||W_q W_q^T – I ||$ during training, to see if the reason of proposed method working well in practice is due to $q(W)$ being fairly close to being orthogonal or not.
• L079: authors claimed SotA results on pMNIST, however they did not compare against other 4-bit sequence-to-sequence models, for instance SSM models such as Mamba [1], transformer models such as LLaMA-3 [2] etc.
• Considered benchmarks are very small by today standards. While most of them seem standard in ORNN literature, it would make a story more convincing if authors included some of the more recent real-world datasets and benchmarks. For instance, it would be insightful to evaluate the proposed methods on some of common reasoning language tasks (MMLU, HellaSwag, Winogrande).

[1] Gu et al., “Mamba: Linear-Time Sequence Modeling with Selective State Spaces”. ArXiV: 2312.00752

[2] Dubey et al., “The Llama 3 Herd of Models”. ArXiV: 2407.21783

• There are awkward and missing citations throughout the paper. For example, the citation for fastGRNN should have been first on line 114 instead of line 135. Besides, I think the representation, such as "activations for LSTM Hou et al. (2017)" in line 105, "Penn TreeBank dataset Marcus et al. (1993) (PTB)" is awkward.
• There are some issues on writing (see minor issues and questions below)

Minor Issues

1. There are some minor points on writing:

• Line 90: “The reasons .. is” -> “The reasons .. are”
• Lines 122 and 321: footnotes are written incorrectly
• Footnote 4: “Sections 3.4” should be revised correctly
• Equation in line 203, footnote 5, caption of Table 3, and line 466: please add a comma to the end of the sentence
• Line 286: add a colon to the next of the bolded sentence
• Subtitle 3.4: “QORNN are” -> “QORNN is” or “QORNNs are”
• Tabels 1 and 2, “fromKiani et al. (2022)” -> “from Kiani et al. (2022)”
• Table 2, “sizes for Copy, MNIST” -> “sizes for Copy-task, sMNIST”

2. Are the words "steps", "power", "timesteps" and "length" the same meaning? Mixed terms can confuse the reader. I recommend revising them for clarity. Other examples include copy/copy-task, SSMs/SSM/SSSM, etc.

• The biggest shortcoming of this paper is the limited novelty. Several points regarding this:
  • The authors combine two off the shelf ORNN training algorithms with the most simple (and arguably outdates, see later) flavor of QAT. In other words, they add $q_k(W_i)$ to these algorithms (cf line 3, 4 in algorithms 1, 2, respectively) and assume STE.
  • While I found the investigation in sec 3.4 interesting (see above), I would have expected that these insights come back in their algorithm design (or at least that the authors evaluate such metrics for their proposed approach, look at question whether these metrics in practice correlate with QORNN performance etc).
• On the quantization side, they seem to miss/ignore a lot of innovation that happened in the last 5ish years which could help to potentially have much better performance at low bit-widths. Most importantly:
  • They argue keeping the scale fixed is common practice (line 241/242). Since the introduction of LSQ [1] this is long not common practice anymore.
  • It is unclear to me why the authors not consider per-channel weights. As per-channel weights still work with full fixed-point/integer arithmetic [2], this is supported by almost any HW and commonly used by most quantization literature in the past years. This adds an additional degree of freedom that might be very helpful for ORNNs as by changing the scale ($\alpha$), as one could ensure that rows (or columns, depending on notation) still have a norm of 1 which seems important (cf sec 3.2).
• The proposed QORNNs are actually not orthogonal as the name or text suggest. Only the latent weights ($W_i$) are orthogonal, but the actual quantized weights used for inference ($q_k(W_i)$) are only approximately orthogonal. As the authors themselves show (cf figure 1, sec 3.4), this doesn’t give any guarantees and could be detrimental.
• As the paper positions itself more as ‘explorative’ (and QAT contribution is very limited), I would expect that they also more closely explore PTQ approaches. There are several degrees of freedom that are unexplored, e.g. setting the correct scale (alpha) or applying/adapting common PTQ algorithms such as AdaRound [3] or GPTQ [4].
• Minor:
  • The experimental evaluation is limited to only ‘toy-sized’ datasets/tasks.
  • While it is nice they obtain bounds for q_min/max, the established bounds are so loose that from a practical perspective such bounds are not useful (nor similar to the earlier study they are used to design or evaluate the algorithm).
  • I do miss a comparison to challenges in quantizing transformers or SSMs. While it is arguable whether comparing to transformers it out of the scope of such a paper (as the authors claim), at least discussing/comparing whether the challenges in ORNNs are similar or different to transformers/SSMs would be helpful (e.g. do they suffer from similar outliers as observed in transformers, cf. [5,6]).
  • Regarding SSMs, there is some recent work that would be interesting to compare to [7, 8].

References:

• [1] Esser et al., ICLR 2020, Learned Step Size Quantization.
• [2] Nagel et al., 2021, A White Paper on Neural Network Quantization.
• [3] Nagel et al. ICML 2020, Up or Down? Adaptive Rounding for Post-Training Quantization.
• [4] Frantar et al., ICLR 2023, Gptq: Accurate post-training quantization for generative pre-trained transformers.
• [5] Bondarenko et al., EMNLP 2021, Understanding and Overcoming the Challenges of Efficient Transformer Quantization.
• [6] Dettmers et al., NeurIPS 2022, LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale.
• [7] Pierre et al., ES-FoMo 2024, Mamba-PTQ: Outlier Channels in Recurrent Large Language Models
• [8] Abreu et al, NGSM 2024, Q-S5: Towards Quantized State Space Models