Unsupervised Sign Language Translation and Generation

Q1：Experimental settings: supervised finetune results and other dataset results.

1.1 Fine-tuning USLNet on parallel corpus Thanks for your suggestion. We conduct experiments and report the experimental results:

The results empirically show that unsupervised training can provide a good representation and significantly improve the supervised translation method (B@4 1.0 --> 1.4).

1.2 Other dataset results For the results on other dataset. Thank you for suggestion. Given that the amount of PHOENIX-2014T and CSL-Daily is too small for training unsupervised model, we do not adopt this results. In the future, we will explore and experiment USLNet on other large sign language dataset, such as OpenASL.

Q2： Align problem in slide window aligner.
Yes, video and glosses are monotonically aligned. However, because BOBSL does not have human-evaluated sentence-level glosses annotations, we suggest that video and text are roughly aligned and align video with text. Morepver, we incorporate validation of BOBSL that video and text are roughly aligned.

To address this issue, we must first obtain the golden sign order. In the sign language domain, text-based interpretations of signs are referred to as glosses[1]. However, BOBSL does not provide sentence-level human-annotated glosses.Therefore, we utilized the automatic gloss annotation released in [2]. This gloss annotation consists of word-level annotations, presented as [video name, global time, gloss, source, confidence]. We converted these gloss annotations into sentence-level annotations and assessed the consistency between the gloss (sign) and text orders.

We will add the experimental results in the revisd version.

[1] Núñez-Marcos A, Perez-de-Viñaspre O, Labaka G. A survey on Sign Language machine translation[J]. Expert Systems with Applications, 2023, 213: 118993.
[2] Momeni L, Bull H, Prajwal K R, et al. Automatic dense annotation of large-vocabulary sign language videos[C]//European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022: 671-690.

Q3: Descriptions problem for the process of the aligner.
Thank you for your advice. We will simplify our illustration of the slide window aligner in the revised version. We will add a discussion of dual learning in the related works.

Q4: Dual Learning introduction.
We appreciate your suggestions regarding dual learning algorithms in our work. Following your recommendation, we will include a subsection discussing dual learning in the revised version. Here, we provide a brief outline of the proposed subsection, which will cover the following points:
Dual Learning. [3] propose dual learning to reduce the requirement on labeled data aiming to train English-to-French and French-to-English translators. It regards that French-to-English translation is the dual task to English-to-French translation. Thus, it designs to set up a dual-learning game which two agents , each of whom only understands one language and can evaluate how likely the translated are natural sentences in targeted language and to what to extent the reconstructed are consistent with the original. Moreover, researchers exploit the duality between two tasks in training[4] and inference [5] stage , so as to achieve better performance. Dual learning algorithms have been proposed for different tasks, such as translation[3], sentence analysis[8], image-image translation[6], image segmentation[7]. USLNet extend dual learning to sign language realm and design dual cross-modality back-translation to learn sign language translation and generation tasks in one unified way.

[3]He D, Xia Y, Qin T, et al. Dual learning for machine translation[J]. Advances in neural information processing systems, 2016, 29.
[4]Xia, Y., Qin, T., Chen, W., Bian, J., Yu, N., and Liu, T.-Y. Dual supervised learning. ICML, 2017a.
[5]Xia, Y., Bian, J., Qin, T., Yu, N., and Liu, T.-Y. Dual inference for machine learning. In Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence (IJCAI), pp. 3112–3118, 2017b.
[6]Yi Z, Zhang H, Tan P, et al. Dualgan: Unsupervised dual learning for image-to-image translation[C]//Proceedings of the IEEE international conference on computer vision. 2017: 2849-2857.
[7]Luo P, Wang G, Lin L, et al. Deep dual learning for semantic image segmentation[C]//Proceedings of the IEEE international conference on computer vision. 2017: 2718-2726.
[8]Xia Y, Tan X, Tian F, et al. Model-level dual learning[C]//International Conference on Machine Learning. PMLR, 2018: 5383-5392.

Q1：Other results on BOBSL and acknowledge that the challenging of BOBSL.
We greatly appreciate your contribution of state-of-the-art (SOTA) results and insights emphasizing the challenges associated with BOBSL. For the work[1], the best model using the context and spotting information achieves 2.88 B@4 points. However, the model can not obtain context and spotting information in our setting. To make a fair comparison, the appropriate baseline for comparison should be B @ 4=1.27 for video-to-text, as depicted in Table 4 in [1], given the fact that USLNet also do not have extra knowledge.
[1] Sincan O M, Camgoz N C, Bowden R. Is context all you need? Scaling Neural Sign Language Translation to Large Domains of Discourse[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023: 1955-1965.

Q2：Experiment on different datasets.
Thank you for suggestion. We follow your suggestion to conduct the experiments and will report the results of Open-ASL in the next few days, given that pretraining and unsupervised training cost lots of time. By the way, because the amount of PHOENIX-2014T is too small for training unsupervised model, we do not adopt this results.

Q3：Unsupervised + Supervised FT.
Supervised Ft Results: unsupervised way to obtain more knowledge representation:

From the above table, we can use unuspervised training way can provide one good representatio and is significant for improve supervised translation method (B@4 1.0 --> 1.4).

Q4: Qualitative results and failure analysis.
Thanks for your suggestions. We completely agree with your opinion that we should dig deep into “why” and “how”. Moreover, it is especially significant for this topic. In the next paragraph, we will give our analysis and possible paths to success.
1 For the reason (“why”) part, we will divide it into results analysis and dataset analysis. first give the analysis about results to tell which we can do relatively better and show comparison between our output and supervised output. 1.1 Case study which can be research on good case, bad case and case for comparison:
Good cases:

Bad cases:

From digging into our results, we find that we can do relatively better in Main ingredients(eg: bus, I, anything), but always fail in other detail, such as proper noun(eg: Ma Effanga), and complex sentence(which is that).

Comparison cases:

From results comparison, we find that we can find that our results are competitive to supervised method. Moreover, in some ways, we can obtain more accurate output(for example, especially).

1.2 Dataset Analysis
As [1] said, the BOBSL dataset is challenging, and that we totally agree with you that it need several breakthroughs to progress in large scale SL translation/generation.

[1]Sincan O M, Camgoz N C, Bowden R. Is context all you need? Scaling Neural Sign Language Translation to Large Domains of Discourse[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023: 1955-1965.

2 For the “how” part, we propose a two-fold approach. First, we suggest allowing unsupervised learning to serve as a representation learning stage. Second, we recommend enhancing USLNet by focusing on improvements in both the pretraining and aligner components.

2.1 Supervised Finetuning Results: unsupervised way to obtain more knowledge representation.

From the above table, we can use unuspervised training way can provide one good representation and is significant for improve supervised translation method (B@4 1.0 --> 1.4).

2.2 Paths to success:

USLNet can be divided into two primary components: the pretraining module (comprising the text pre-training module and the video pre-training module) and the mapper part (slide window aligner). Consequently, the paths to success can be categorized into two aspects. The first aspect involves pre-training, where we can adapt our method using multi-modal models, such as Video-Llama. The second aspect focuses on designing an effective mapper. We will add the discussion in the revised version.

Q1: Analysis of the results and possible path to success.

We appreciate your suggestions and agree with your perspective on delving deeper into the "why" and "how" aspects, particularly for this topic. In the following paragraph, we will provide our analysis and potential paths to success.

1 Regarding the "why" aspect, we will conduct a thorough analysis of the results, identifying the areas in which our approach performs well and those that require further improvement.

1.1 Case study which can be research on good case, bad case and case for comparison:
Good cases:

Bad cases:

From digging into our results, we find that we can do relatively better in Main ingredients(eg: bus, I, anything), but always fail in other detail, such as proper noun(eg: Ma Effanga), and complex sentence(which is that).

Comparison cases:

From the results, we observe that our outcomes are competitive with those of supervised methods. Furthermore, in certain instances, we can achieve more accurate output (for example, particularly in specific cases).

2 Regarding the "how" aspect, we propose a two-fold approach. First, we suggest allowing unsupervised learning to serve as a representation learning stage. Second, we recommend enhancing USLNet by focusing on improvements in both the pretraining and aligner components.

2.1 Supervised Finetuning Results: unsupervised way to obtain more knowledge representation.

From the above table, we can use unuspervised training way can provide one good representation and is significant for improve supervised translation method (B@4 1.0 --> 1.4).

2.2 Paths to success:

USLNet can be divided into two primary components: the pretraining module (comprising the text pre-training module and the video pre-training module) and the mapper part (slide window aligner). Consequently, the paths to success can be categorized into two aspects. The first aspect involves pre-training, where we can adapt our method using multi-modal models, such as Video-Llama. The second aspect focuses on designing an effective mapper. We will add the discussion in the revised version.

Q2: Different resolution:

We have conducted experiments using higher resolution; however, the results do not indicate any performance improvement. A possible explanation for this outcome could be that the BOBSL dataset is inherently challenging[1], and increasing the video resolution may further complicate the task. We will add the discussion in the revised version.

[1]Sincan O M, Camgoz N C, Bowden R. Is context all you need? Scaling Neural Sign Language Translation to Large Domains of Discourse[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023: 1955-1965.

Q3: Pre-training for text and video model:
In our experiments, we have implemented text and video pretraining to incorporate additional knowledge and utilize it for initialization (Section 2.1 and 2.2). Moreover. Pretraining is indispensable for USLNet.

Q4: Order validation:
Thank you for your suggestion. Indeed, incorporating validation of BOBSL can make the sliding window aligner more convincing. To address this issue, we must first obtain the golden sign order. In the sign language domain, text-based interpretations of signs are referred to as glosses[2]. However, BOBSL does not provide sentence-level human-annotated glosses.

Therefore, we utilized the automatic gloss annotation released in [3]. This gloss annotation consists of word-level annotations, presented as [video name, global time, gloss, source, confidence]. We converted these gloss annotations into sentence-level annotations and assessed the consistency between the gloss (sign) and text orders.

We will add the experimental results in the revisd version.

[2] Núñez-Marcos A, Perez-de-Viñaspre O, Labaka G. A survey on Sign Language machine translation[J]. Expert Systems with Applications, 2023, 213: 118993.
[3] Momeni L, Bull H, Prajwal K R, et al. Automatic dense annotation of large-vocabulary sign language videos[C]//European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022: 671-690.

Thanks for the detailed response. After looking through the details and reading other reviews/rebuttals I am unfortunately keeping my ratings.

Q1：Discussion text-to-video aligner algorithms.
We appreciate your suggestions regarding the text-to-video aligner algorithms in our work. Following your recommendation, we will include a subsection discussing text-to-video aligners in the revised version.

Here, we provide a brief outline of the proposed subsection, which will cover the following points:
a) Text-to-video aligners in sign language domain can be broadly classified into two main categories. The first category involves the use of animated avatars to generate sign language, relying on a predefined text-sign dictionary that converts text phrases into sign pose sequences [1-3]. The second category encompasses deep learning approaches applied to text-video mapping. [4-5] adapt the transformer architecture to the text-video domain and employ a linear embedding layer to map the visual embedding into the corresponding space. Unlike these methods, which can only decode pose images, our Unsupervised Sequence Learning Network (USLNet) is capable of generating videos. We address the length and dimension mismatch issues by utilizing a simple sliding window aligner.

b) Text-to-video aligners in other domains have also been proposed. [6] introduced a method for automatic redubbing of videos by leveraging the many-to-many mapping of phoneme sequences to lip movements, modeled as dynamic visemes. The Text2Video approach [7] employs a phoneme-to-pose dictionary to generate key poses and high-quality videos from phoneme-poses. This phoneme-pose dictionary can be considered as a token-token mapper. Analogously, USLNet quantizes discrete videos and extracts video tokens, a standard technique in the audio domain [8-10]. Consequently, the sliding window aligner also serves as a token-token aligner. However, unlike the Text2Video method, which performs a lookup action to obtain target tokens, our approach decodes the target token using all source tokens.

[1]Glauert, J., Elliott, R., Cox, S., Tryggvason, J., Sheard, M.: VANESSA: A System for Communication between Deaf and Hearing People. Technology and Disability (2006)
[2]Karpouzis, K., Caridakis, G., Fotinea, S.E., Efthimiou, E.: Educational Resources and Implementation of a Greek Sign Language Synthesis Architecture. Computers & Education (2007)
[3]McDonald, J., Wolfe, R., Schnepp, J., Hochgesang, J., Jamrozik, D.G., Stumbo, M., Berke, L., Bialek, M., Thomas, F.: Automated Technique for Real-Time Production of Lifelike Animations of American Sign Language. Universal Access in the Information Society (UAIS) (2016)
[4]Saunders B, Camgoz N C, Bowden R. Progressive transformers for end-to-end sign language production[C]//Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XI 16. Springer International Publishing, 2020: 687-705.
[5] Saunders B, Camgoz N C, Bowden R. Adversarial training for multi-channel sign language production[J]. arXiv preprint arXiv:2008.12405, 2020.
[6]Sarah L Taylor, Moshe Mahler, Barry-John Theobald, and Iain Matthews, “Dynamic units of visual speech,” in Proceedings of the 11th ACM SIGGRAPH/Eurographics conference on Computer Animation, 2012, pp. 275–284.
[7]Zhang, Sibo, Jiahong Yuan, Miao Liao, and Liangjun Zhang. "Text2video: Text-driven talking-head video synthesis with personalized phoneme-pose dictionary." In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 2659-2663. IEEE, 2022.
[8]Hsu W N, Bolte B, Tsai Y H H, et al. Hubert: Self-supervised speech representation learning by masked prediction of hidden units[J]. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2021, 29: 3451-3460.
[9]Wang C, Chen S, Wu Y, et al. Neural codec language models are zero-shot text to speech synthesizers[J]. arXiv preprint arXiv:2301.02111, 2023.
[10]Borsos Z, Marinier R, Vincent D, et al. Audiolm: a language modeling approach to audio generation[J]. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2023.

Q2: Similarities and differences between back-translation and reconstruction loss.
Yes, the loss functions employed in our UNSLT exhibit a high degree of similarity. The distinctions between these two losses stem from the modality of the intermediate representations. In the case of back translation (video-text-video), the intermediate representation is a cross-modal (text) representation, which serves to align the cross-modal representations. Conversely, the intermediate representation for video reconstruction is an unimodal (video) representation, primarily utilized for learning the reconstruction of video sequences.