Sign2GPT: Leveraging Large Language Models for Gloss-Free Sign Language Translation

Thanks for the review, and your insightful comments.

W1: Using pretrained language model to boost sign language translation (SLT) is not surprisingly novel. Several works [1,2] have already verified that pretrained language model can boost (gloss-free) SLT.

While we acknowledge previous research on leveraging pretrained language models for sign language translation, our approach offers several advantages over existing methods.

Chen et al. [1] primarily focus on utilizing manually annotated sign-ordered gloss supervision, relying on Gloss2Text for progressive pretraining on the encoder-decoder language model. In contrast, our approach does not rely on manually annotated glosses. Instead, we concentrate on developing a method for gloss-free sign representation, utilizing a decoder-only language model without the need for progressive pretraining.

Similarly, Zhou et al. [2] have explored gloss-free pretraining, but their approach involves updating the visual encoder and text decoder simultaneously. In contrast, our pseudo-gloss pretraining is independent of the language model. We keep the pretrained language model frozen by leveraging adapters, allowing the use of large-scale pretrained language models. Our architectural design outperforms their approach, as demonstrated in our results.

W2: The name of pseudo-gloss pretraining is a bit confusing, although the method itself is sound. Because the pseudo-glosses are in spoken language order, it is not quite appropriate to call them "glosses".

We termed them pseudo-glosses as they are potential glosses that are signed. However, while we extract them from the spoken language sentence our proposed pretraining is invariant to the order of the pseudo-glosses. As demonstrated in Figures 4, 5, and 6 the pretraining is also able to localize the pseudo-glosses and thus provide them in sign order. We are happy to change them to something more appropriate and are open to suggestions.

W3: The notations of "P", "F", "D" in Table 2 and 3 are also confusing. For example, D denotes without pretraining and P denotes pretraining, then what does "P+D" mean? I suggest authors adding a "check mark" column to show which parts are pretrained.

We have updated our tables shown below. In the table Sign2GPT without Pseudo-Gloss Pretraining is denoted as Sign2GPT and Sign2GPT(w/ PGP) indicates with Pseudo-Gloss Pretraining (PGP). This updated table clearly shows that our approach outperforms the previous state-of-the-art results. We also included Sign($Z$)2GPT(w/PGP) indicating we are giving the extracted features $Z$ to GPT, which consists of a frozen spatial and sign encoder model that has been trained with PGP, demonstrating the strength of our PGP as feature extractors for sign translation.

Translation Results on Phoenix14T:

Translation Results on CSL-Daily:

Thanks for the review, and your insightful comments.

Q1: It's great that the proposed method outperforms GFSLT-VLP. However, GFSLT-VLP is based on MBart with ~600M parameters, and this study adopts XGLM with ~1.7B parameters, making the fairness of the comparison questionable. It's unclear whether the improvements are really from the proposed modeling and pretraining strategy. Could you please add further ablations regarding the size of GPT models?

While we acknowledge the substantial difference in parameter count, a key strength of our approach lies in the frozen pretrained language and vision model, with weight updates facilitated through adapters. This results in a total trainable parameter count of under 17 million, significantly less than GFSLT-VLP's ~600M parameters.

To address the concern about the total parameter count, we conducted an ablation study using a smaller 564M parameter XGLM model. The results, presented in the tables below, reveal a marginal performance reduction on the test set of the Phoenix14T dataset. Even with similar total parameter counts, these results outperform GFSLT-VLP.

Q2: The sign encoder and pseudo-gloss pretraining take a crucial role in the proposed method, but analysis and ablation regarding them are insufficient. Do we need the sign encoder? What if dropping it? Apart from the visual analysis, what about the top-N prediction accuracy and recall?

Thank you for your suggestion about quantitivate analysis for pretraining we will also include the below table of the precision, recall and F1-score for Phoenix14T and CSL-Daily pseudo-gloss.

To address the question about needing a sign encoder we have conducted the ablation study showing the pretraining precision, recall and F1-scores on the extracted pseudo-glosses shown in the table below. The role of the sign encoder forms an important part of the model as it learns temporal features which is reflected by the significant improvement in F1-score.

Q3: Please also add gloss-based SOTA systems in Tables 2 and 3 so that readers can understand the gap.

Thank you for your feedback, we will be updating the paper by adding gloss-based SOTA systems in the relevant tables.

Q4: In Eq (1), you mentioned that V represents keys from textual features while K originates from sign features. Shouldn't they both come from sign features?

Thanks for pointing that out, you are correct that V and K both originate from sign features, we will clarify this in the manuscript.

Q5: How did you set the rank in LoRA?

During our experiments, we set the LoRA rank and alpha values both as 4.

Q6: While Phoenix14T and CSL-Daily are popular, they are less significant due to limited vocabulary and data size. Please consider adding results for DGS3-T (Zhang et al., 2023).

Current benchmarks for state-of-the-art gloss-free results primarily focus on Phoenix14T and CSL-Daily. We appreciate the suggestion to include results for DGS3-T. However, we encountered challenges in obtaining the dataset as the GitHub instructions for downloading are currently non-functional.

To address this, we have contacted the authors of DGS3-T to explore the feasibility of obtaining the dataset. If successful, we intend to include the results as baselines for gloss-free sign translation, in the appendices in the CRC.

The additional analyses, comparisons, and updates, will be incorporated into the revised paper, once the reviewer finds these inclusions satisfactory.

Thanks for the review, and your insightful comments.

W1: The method is mostly based on existing models such as GPT, Dino-V2 and LoRA, so there is not much novelty from the architectural standpoint.

While we have used existing models, we introduce a novel technique and model architecture to break the reliance on gloss which is something that everyone working in sign translation would be interested in. Moreover, our paper highlights technical considerations inherent to sign language translation, including the handling of numerous frames leading to memory constraints and the challenge posed by limited dataset size which we address with our architectural design of using frozen pretrained models with adapters. We also introduce a pretraining strategy specific to our architecture, enabling the learning of pseudo-gloss localization which serves as a crucial representation for the LLM.

W2: There is no ablations to demonstrate if the choice to use parts-of-speech tagging effective. How does it compare to using the words as tokens? Can the downstream translation learn to generate meaningful words that are missing in the parts-of-speech tagging?

The downstream translation effectively learns to replace the missing words as demonstrated by our qualitative translation examples in Tables 5 and 6. This is a key benefit of a strong language model.

Regarding the ablation of parts-of-speech (POS) tagging, we deliberately excluded certain parts of speech as we know they are not represented in sign, e.g. connected words such as 'and' and articles such as 'the' lack a functional sign equivalent. In the table below, we illustrate the impact of utilizing POS on precision, recall, and F1-score for our pretraining results on Phoenix14T with a threshold of 0.2. Notably, when all words are used as tokens, recall significantly decreases from 0.39 to 0.28. This result validates our assertion that not all words have corresponding signs in sign language. As outlined in the paper, this pretraining strategy opens avenues for future research, particularly in areas like sign spotting.

The additional analyses, comparisons, and updates, will be incorporated into the revised paper, once the reviewer finds these inclusions satisfactory.

Thanks for the additional information. I maintain the view that I learn towards acceptance.

Thanks for the review, and your insightful comments.

W1: The introduction part seems inconsistent with the title. The introduction mentions the both large-scale vision and language model, while the title only mentions the language one. What do the authors want to emphasize?

We appreciate the reviewer's careful examination of the introduction and title. Our emphasis on 'leveraging large language models' in the title is intentional. Our goal is to create a sign model/representation that is passed to / leveraged by the GPT model for sign language translation, hence the name Sign2GPT. The 'Sign' component that is leveraged by the GPT model includes the use of a large vision model and our novel pretraining strategy. However, we are more than happy to change the title to address this issue.

W2: The pseudo-gloss pretraining technique shares the similar core idea with CSGCR. The authors should discuss the difference.

We appreciate the reviewer's insights into CSGCR [1] and observe these distinctions between our proposed pseudo-gloss pretraining technique and CSGCR:

• Training Approach - CSGCR employs separate transformer encoders for each vocabulary item, necessitating positive and negative samples during training. In contrast, our method utilizes a single transformer encoder for all pseudo-glosses, ensuring scalability for larger vocabularies.
• Localization Capabilities - Our pretraining strategy exhibits inherent explicit localization capabilities, as presented in Figures 4, 5, and 6.
• Translation Process - Unlike CSGCR's two-step process involving Video2Text (Word Existence Verification) and then Text2Text (Sentence Generation), our approach directly utilizes the pretrained model for Video2Text using the GPT model.

We will update the paper to discuss these differences in the related work.

W3: Utilization of pretrained model is not new. The author should cite the following work [2] and discuss the difference with it.

Thanks for highlighting the reference. We will incorporate a discussion of its relevance in the related work section. To outline the distinctions, Chen et al. [2] focus on utilizing manually annotated sign-ordered gloss supervision, relying on Gloss2Text for progressive pretraining on the encoder-decoder language model. In contrast, our approach has a significant advantage of not depending on manually annotated glosses. Instead, we concentrate on developing a method for gloss-free sign representation, leveraging a decoder-only language model without the need for progressive pretraining of the language model.