Expanding Expressivity in Transformer Models with MöbiusAttention

1. The paper's motivation lacks rigorous justification and empirical support. Specifically, in the second paragraph of the introduction, the authors make several unsupported claims about the limitations of linear transformations and softmax in Transformer attention mechanisms:

   a) The authors assert that "predominantly linear operations restrict the ability of models to capture complex linguistic dependencies," but provide no theoretical analysis or experimental evidence demonstrating these supposed limitations.

   b) The authors claim that linear operations lead to "potential information loss within each attention layer". However, they don't elaborate on what type of information is lost or why this loss occurs.

2. The experimental results raise significant concerns about the practical value of the proposed MöbiusAttention mechanism. While the authors introduce a mathematically sophisticated approach using Möbius transformations, the empirical results show that MöbiusAttention-based BERT performs comparably to the original BERT model, with no significant improvements in performance.

• The paper’s main argument is not persuasive. This paper argues MöbiusAttention can enhance expressivity based on its non-linear operation, Möbius transformation. However, I guess Möbius transformation would not bring significant expressivity enhancement because it is constrained to be projective linear group (PGL) which is a scaled general linear group.

• Experiment results do not support the expressivity argument. First, despite of the enhanced expressivity (in hypothesis), the performance improvement in the GLUE task is not superior (around 0.2 point improvement on average). Second, the argument of overfitting problem due to the enhanced expressivity (in the paragraph, lines 360~370) needs more clarification.

• “Learning to Forget”: the argument “Fig. 8 in the Appendix, vanilla attention almost never assigns zero attention score to a token pair. In contrast, MöbiusAttention gives most of the pairs zero score and only a few a non-zero one” seems not supported by the results. In Figure 8, the average number of zero elements in Vanilla head (af) is 11.17, while Möbius head (gl) is 9.67. More importantly, the attention matrix of MöbiusAttention seems almost uniform.

• “Memory and Time Complexity”: the comparison of parameters (104M vs 110M) seems negligible.

• No experimental comparison between other comparable works. The previous works, such as RoPE and NeuralAttention are not compared with MöbiusAttention.

Minor:

• In ‘Geometric Interpretation’ section, the orientation could be changed in ‘det M_{qj} = 1’ case, since orientation can be changed without changing the volume.
• In lines 314~317 “Möbius transformations offer a robust framework for analyzing and interpreting text”, this argument seems not verified by experiments. The authors might want to view the geometric shapes of some semantically similar text tokens.
• Even though this paper argues that MöbiusAttention does not increase computational complexity significantly, it is not experimentally compared. Due to the separated operations (layernorm, attention) for real and complex parts, it might increase computational cost.
• In Table 2, the ‘Overall’ scores seem not in fair comparison because the number of candidates that contribute to the maximum case selection are different, 2 candidates for baselines and 4 candidates for Möbius.

• Motivation is not clear. I understand the intuition that Möbius transformations can capture more complex patterns and therefore improve the attention. However, I am not sure why we need to capture these patterns and what kind of tasks require these patterns.
• Some statements are inaccurate. For example:
  • line 91: "integrating MöbiusAttention can lead to improved performance across various NLP tasks without necessarily increasing the model size."
  • line 1285: "However, our results (Section 5) show that even with fewer layers, MöbiusAttention leads to performance improvements on many tasks."
  • line 460: "As seen in the attention heatmaps in Fig. 8 in the Appendix, vanilla attention almost never assigns zero attention score to a token pair. In contrast, MöbiusAttention gives most of the pairs zero score and only a few a non-zero one."
  • These statements can not be drawn from experimental results.
• The experiments also have several limitations as follows:
  • Performance is weak: The performance improvements brought by MÖBIUSATTENTION are marginal. For example, from Table 2, we can see MobRoFormer H & T can only achieve 0.27 performance improvement compared to RoFormer but consuming more parameters. Given this improvement is marginal, It is not clear that whether the performance improvement is significant. Therefore, I recommend the authors to test this.
  • Lack of Efficiency Analysis: Intuitively, enabling attention to capture more complex patterns will lead to an increase in FLOPs and inference latency. However, the authors do not provide analysis in these aspects.
  • Lack of Large Scale Experiments: Currently, the authors only mainly focused on small models and GLEU tasks. It is necessary to verify the effectiveness of MöbiusAttention in models with larger sizes and broader tasks.

The results just don't seem to be particularly strong. The level of improvement seen could just be from doing lots of ablations and model retrains with different seeds.