Geo-Sign: Hyperbolic Contrastive Regularisation for Geometrically Aware Sign Language Translation

We thank the reviewer for their constructive comments. We have conducted several new experiments since our submission to directly address their concerns.

1. New Experiments on Other Datasets (Weakness 1)

To address the primary concern about the limited evaluation, we are pleased to report new results on two widely-used ASL benchmarks: How2Sign (for SLT) and WLASL2000 (for ISLR).

How2Sign (SLT):

Our method shows a clear improvement over the baseline, confirming its benefits generalise to a new language and a more challenging translation task.

WLASL2000 (ISLR):

On the recognition task, our pose-only method surpasses both the pose-only and the stronger multi-modal (Pose+RGB) baselines.

2. Deeper Ablation on Geometric Properties (Weakness 2)

To provide a better understanding of how the geometric properties are working, we conducted a new ablation study on pose noise robustness. We added Gaussian noise to the final pose representations and measured the impact on performance. The results show that our hyperbolic model is significantly more robust than the baseline, demonstrating a tangible benefit of the learned geometric structure.

We also direct the reviewer to the supplementary material where we include several key analyses on geometric structure. Specifically, Figure 3 (Supp) offers a direct visualization of the learned representation space, demonstrating a clear hierarchical structure where fine-grained hand/face features are pushed to the high-curvature periphery while coarse body features remain central. This visual structure is supported quantitatively by Figure 2 (Supp), which plots the geodesic distances of embeddings from the origin, showing that the model consistently learns to place hand features further into the manifold to leverage its greater representational capacity.

3. On the Interpretation of the Attention Heatmap and Euclidean Performance (Question 1)

The reviewer's intuition is correct: the sign for “吃面条” (chī miàntiáo, 'eating noodles') involves coordinated motion of the dominant hand (mimicking chopsticks) and the face/mouth area. The Euclidean model's heatmap, however, shows diffuse attention. We presume that it still provides an improvement over the baseline by learning a "brute-force" correlation rather than identifying the most salient features. This would also explain why the Euclidean Token version actually performs worse than the Pooled version on ROUGE, because the fine-grained alignment required by the Token method can be detrimental when the feature space is not sufficiently discriminative. The attention method however, also introduces a small number of additional trainable parameters (around 1M), and these may be helping to regularise the model improving on the other n-gram metrics. We will add a discussion of this result to the paper and include additional attention heatmaps to the supplementary for review.

Finally, we will correct the "sign-language" typo in the abstract in the final version.

We hope these new results and clarifications address all the concerns and we are happy to answer any further questions or implement any suggestions the reviewer may have. Thanks again for the review and we look forward to the discussion period.

We thank the reviewer for their positive feedback on our work's novelty and for the constructive comments.

1. On Clarity, Framing, and Motivation (Weakness 1, Question 2)

We agree that a clearer high-level overview would improve the paper's readability. In the revised version, we will add a new introductory paragraph to Section 3 (Methodology* that provides a step-by-step walkthrough of the Geo-Sign framework before delving into the geometric details.

To clarify the motivation for our two alignment strategies: we designed them as a deliberate comparative study. The "Pooled" method tests the baseline hypothesis of aligning global, sentence-level semantics. The "Token" method tests a more complex hypothesis, i.e that a fine-grained, attention-based alignment between individual pose parts and text tokens would yield superior results. We will explicitly state this motivation in Section 3.3. Furthermore in the supplementary material we provide a detailed analysis showing that the pooled method incurs less latency at a small cost in performance. We can incorporate this into the main paper to make the motivation for both approaches clearer and improve the flow of this section.

2. On Generalization and Evaluation on Other Datasets (Weakness 2, Question 1)

To address your concern (and other reviewers) about generalisation, we have, since our submission, evaluated our method on two additional, widely-used ASL benchmarks: How2Sign (for SLT) and WLASL2000 (for ISLR).

How2Sign (SLT): Our method demonstrates an improvement over the Uni-Sign pose-only baseline, confirming its benefits generalize to American Sign Language.

WLASL2000 (ISLR): On the recognition task, our pose-only method not only improves upon the pose-only baseline but also surpasses the stronger multi-modal (Pose+RGB) version of Uni-Sign.

These new results provide strong evidence that our approach is not limited to a single dataset and generalises effectively across different languages and tasks.

3. On the Classification of Prior Art (CV-SLT) (Weakness 3, Question 3)

This is an interesting discussion point. We would like to note that in the official repository and methodology it can be seen that their code uses visual features extracted from a model pre-trained on a sign-to-gloss (s2g) task (specifically, the pre-trained embeddings from MMTLB, which are themselves from a gloss-prediction network).

While the CV-SLT authors state they do not use "additional gloss supervision during the SLT training," we could argue that their method is, by definition, gloss-based. The visual features are not learned end-to-end from video-text pairs; they use prior gloss-level supervision. This is a bit different from truly gloss-free methods like ours, which learn representations without any exposure to gloss annotations at any stage. We do not feel that strongly about moving it in the table (i.e putting it with gloss free) so we will let the reviewer decide what they prefer given this context. Furthermore the architecture uses RGB features which are not directly comparable with our pose-only approach.

Thanks again for the nice review and we look forward to the discussion period.

We thank the reviewer for their constructive and detailed review. We agree with the weaknesses presented and have addressed them below.

Addressing Weaknesses

1. On Task Suitability (SLT vs. SLR)

The reviewer raises a very fair point about the suitability of SLT for evaluating embedding quality, and we agree that an evaluation on ISLR is missing from the paper. As discussed in our other rebuttals, since our initial submission, we have evaluated our method on the WLASL2000 benchmark for ISLR. We are pleased to report that our method outperforms not only the pose-only baseline but also the stronger multi-modal (Pose+RGB) variant of Uni-Sign.

We will add the full table of results and additional details to the full paper or can post here during the discussion period if requested.

To address concerns about the limited dataset scope and potential dependence on CSL-Daily's characteristics, we have conducted new experiments on the How2Sign dataset for ASL-to-English translation.

As the reviewer correctly notes, How2Sign has a wider semantic gap and fewer repeated sentences. The fact that our method still provides a significant improvement on this more challenging dataset provides strong evidence that our results are not an artifact of CSL-Daily's structure and generalizes well.

2 Sampling in InfoNCE

To clarify, our work uses the standard in-batch sampling protocol of InfoNCE-style loss presented in Equation (5). There is no separate or special sampling step.

For any given anchor pose embedding $\mathbf{p}_i$ from a training batch of size $B$, its corresponding text embedding $\mathbf{t}_i$ is treated as the sole positive sample. The other $B - 1$ text embeddings in that same batch, $\{\mathbf{t}_j\}_{j \neq i}$, automatically serve as the in-batch negatives.

This objective, a softmax cross-entropy over the geodesic distances, requires the model to correctly identify the positive pair $(\mathbf{p}_i, \mathbf{t}_i)$ from the set of all $B$ candidates within the batch. Minimizing this loss directly forces the model to decrease the distance between corresponding pairs while simultaneously increasing the distance to all other non-corresponding pairs in the batch.

3. On Dependency on Pose Estimation Quality

The reviewer was not the only one to bring up the dependency on pose estimation quality. To quantify this risk, we conducted a new experiment where we added Gaussian noise to the pose representations. The results show that our hyperbolic method is significantly more robust and degrades more gracefully than the baseline.

As discussed with the other reviewers, we believe this is because the learned hyperbolic geometry creates larger decision margins between sign classes, making the representations more resilient to perturbations. We will add this full analysis to the appendix.

4. On Limited Dataset Scope (Addressed in Point 2)

As shown above, our new results on How2Sign and WLASL provide examples of our method's cross-linguistic and cross-task generalisability.

5. On Unclear Notation in Figure 1

Thank you for pointing this out; we agree it is unclear. We will revise the caption for Figure 1 in the final version to explicitly define all symbols.

We are confident that these new experiments address the weaknesses raised and look forward to the discussion period. Thanks again for taking the time to review the paper and giving some nice feedback and comments to improve the paper.

We thank the reviewer for their thoughtful and constructive feedback. We are happy to address the weaknesses and questions below.

Addressing Weaknesses

1. Generalization to Other Datasets and Tasks

As requested, we provide additional results on How2Sign (ASL Translation) and WLASL2000 (ASL Recognition).

How2Sign (SLT)

On this translation task, our geometric regularization provides a clear improvement over the Uni-Sign baseline, demonstrating generalization to a new language (ASL) and data domain.

WLASL2000 (ISLR)

On the recognition task, our pose-only method not only improves upon the pose-only baseline but also surpasses the stronger multi-modal (Pose+RGB) variant of Uni-Sign demonstrating that the method is adaptable to recognition as well as translation tasks.

We have only included the baseline (Uni-Sign) as reference here for brevity but can provide the full table on request. We will add these results and some discussion to the final paper.

2. Robustness to Pose Noise and Non-Manual Features

The reviewer raises a great point regarding the importance of non-manual features (NMFs) that can only be captured accurately with RGB data and the challenges of pose-only methods. This work is explicitly framed to address privacy-aware SLT and so we don't consider RGB features, however we do include 16 keypoints for the face which should capture some of this information. To empirically address the concern of in-the-wild poses, we conducted a new ablation study, adding Gaussian noise to the pose representations. The results demonstrate that our hyperbolic method is substantially more robust to noise than the baseline, maintaining higher performance and degrading more gracefully. We will add these results as an additional ablation in the paper.

We hypothesize this increased robustness stems from the geometric properties of the Poincaré ball. The exponential volume growth forces a sparser representation, creating larger geodesic margins between distinct sign classes, making the model less susceptible to noisy perturbations.

3. Computational Cost

The reviewer is correct that our hyperbolic branch introduces computational overhead during training, a point we address in detail in Appendix F.1. As discussed, this latency is partly due to the necessity of using FP32 for hyperbolic operations to maintain numerical stability. However, this is a one-time training cost. Crucially, at inference time, the overhead is zero, as the regularization branch is removed.

4. Statistical Significance

We can only agree with the reviewer, though we note it is a limitation shared by all prior work in this area due to the computational requirements for training these large SLT models. To ensure fairness and reproducibility, we used the same seed and hyperparameters as the baseline. We also include a detailed overview of all hyperparameters in the supplementary material. The code for all experiments and weights are also available for both our pretrained and fine-tuned models.

5. Sensitivity of the Weighted Fréchet Mean

As discussed in Appendix D.1 (Proposition D.1), the computation is highly stable. As we discuss in the supplementary, the Poincaré ball is a Hadamard manifold, where the squared geodesic distance is a strictly convex function. This guarantees that a unique Fréchet mean exists and that our iterative algorithm (Algorithm 1) is guaranteed to converge to it, irrespective of initialization. We are happy to move this into the main paper at the reviewers request.

Questions

1. Generalization to other sign languages

As demonstrated in our new experiments on How2Sign (ASL) and WLASL2000 (ASL), our method generalizes well. We hypothesize this is because the hierarchical nature of kinematics (torse $\\to$ arm $\\to$ hand) is a fundamental principle in sign languages. Our model, by using a geometry inherently suited to such hierarchies, learns this compositional structure, allowing it to adapt across different languages.

2. Dynamic Curvature

This is an interesting idea! We did experiment with learning distinct curvature parameters for each body part but, as the reviewer might anticipate, we found it introduced instability into the Riemannian optimization. It is technically feasible to have adaptable curvature over the temporal dimension, perhaps adapting it based on kinematic properties, but again challenging to optimize.

3. Simpler Attention Variants

Our choice of Möbius transformations was principled, as they are the formal isometries of the Poincaré ball and thus the geometrically sound analogue to the affine transformations critical to standard Euclidean attention. This enables a dynamic, learned alignment, which we argue is crucial for the complex alignment task in SLT. However, we agree that a simpler distance based metric would be a nice ablation to the paper. Unfortunately we don't have the resources during the rebuttal period, but will perform an additional ablation for the final version of the paper.

4. Gradient Stability

The reviewer correctly identifies this challenge, which we address in the paper. As detailed in Section 3.4 and expanded in Appendix G.2, we employ two primary safeguards: 1) tangent vector clipping before the exponential map and 2) maintaining FP32 precision for all geometric operations. These steps have proven effective in ensuring stable training.

Thanks again for the nice review, we look forward to the discussion period.

Since the initial rebuttal we have been able to run the additional requested ablation experiment on the simpler attention methods.

Regarding the dot-product variant, as discussed in our initial comment, a standard Euclidean dot product is geometrically inconsistent with the Poincaré manifold. However, we implemented the distance-only variant where attention scores are derived from the negative geodesic distance on the hyperbolic manifold. The ablation was performed on CSL-Daily with the same training and hyper-parameter setup as the main experiment.

The distance-only model's n-gram precision (BLEU-4) offers no improvement over the Euclidean baseline. Interestingly, we see some improvement on the content recall (ROUGE-L) performance.

We attribute this to the hyperbolic space's greater capacity for feature separation, which aids a simple retrieval mechanism in recalling a more accurate set of semantic concepts (improving ROUGE-L). However, arranging these concepts into high-precision phrases requires a more sophisticated, learnable mechanism to model syntactic relationships which is provided by the learnable Möbius operations.

This ablation confirms that the method benefits from the principled combination of the hyperbolic geometry and the learnable Möbius attention.

This analysis has been added to the paper's appendix. Again, we appreciate the feedback, which has led to a more robust validation of our work.