SignKD: Multi-modal Hierarchical Knowledge Distillation for Continuous Sign Language Recognition

[Major]

• Although the overall flow of the technical design is plausible, the proposed SignKD framework combines two existing components well-established in literature (i.e., multi-modal fusion layer and feature-level knowledge distillation), which is not an astounding result and can weaken the technical novelty of the work. As submitted to a top-tier ML conference, further theoretical or intuitive insight on the architectural design would be expected.

• The claim for the state-of-the-art performance in CSLR tasks needs to be reconsidered. Specifically for the video-only methods, is the video-based backbone S3D pretrained on video data, or initialized from scratch? While the most performant baseline SEN (Hu et al., 2023c) is based on the architecture with frame-level 2D CNN followed by LSTM, but the proposed work adopts a strong video backbone. Such architectural differences and the following advantages should be acknowledged when comparing performances. Can the proposed work still achieve superior performances even when built upon such frame-level architecture? In addition, for the multi-modal-based experiments, comparison with baseline methods in the same modality condition is missing. For example, how much the performance drop would be observed when the same kind of modalities with baseline methods (i.e., video+pose) are used for training the final multi-modal teacher model?

[Minor]

• For the additional techniques explained in Appendix C. in detail, missing citation for the sign pyramid network [Chen et al., 2022]. In addition, frame-level supervision has been also previously explored [Min et al., 2021, Hao et al., 2021, Jang et al., 2023]. Further discussions on advantages or differences of the proposed work compared to previous works would be more helpful for potential readers.

• Computational aspects for the proposed model are missing. Batch size during training is not provided. In addition, as the training cost for a CSLR model is highly demanding, it would be beneficial to share total training hours and required VRAM memory. Moreover, how many frames the final student model can process during inference (i.e., FPS)? Considering real-world deployment, reporting such quantities and comparing with existing CSLR methods is suggested.

• For the ablation results from Table 4(b), one additional row corresponding to the result where only CTC and O are marked is expected. This can be compared with the item where all the feature-level distillation losses are applied (the last row).

• Related to the reproducibility, it is unclear whether the codes for training and inference would be released. If there is no plan to open the codes, could you provide a brief reason?

1. My major concern is the novelty. Since TwoStream Network (neurips 2022) already proposed to jointly model videos and keypoints, adding another modality, i.e., optical flow, is somehow incremetal. Many techiniques are directly borrowed from TwoSream Network, including how to represent keypoints, joint head network, and sign pyramid network. Also, jointly modeling the three modalities has already appeared in a relevant literature [1] (although [1] is for isolated recognition, it is trivial to extend to continuous recognition by just replacing the cross-entropy loss with the CTC loss), and thus the authors' claim that they are the first to utilize the three modalities is weak. The DNF cited in the paper also proposes to model videos and optical flow.

2. Compare to some purely video-based methods (without distillation), e.g., CorrNet and CVT-SLR, the performance gain is small as shown in Table 1.

3. Given that it is a empirical paper, similar to TwoStream Network, more comprehensive evaluation including sign language translation shall be considered.

4. What does "hierarchical" mean? The authors distill features in a stage-to-stage way. Then where is "hierarchical"? The authors also claim that "this involves transferring knowledge from the shallow layers to the deep layers" in page 6, but I cannot find which techinique is corresponding to this claim.

5. Why the authors choose a somehow old algorithm to extract optical flow? Some new methods, e.g., RAFT, shall be considered.

Generally, only the distillation part of the paper is relatively new, but I don't think it is enough for ICLR. As a more empirical paper, I suggest the authors to further evaluate their method on sign language translation.

[1] Skeleton Aware Multi-modal Sign Language Recognition, CVPRW 2021

1. The core idea is simple and limited novel. The utilization of different modalities is widely validated in many general computer vision tasks, i.e., human action recognition, action temporal localization, etc. Directly stacking multiple modalities to enhance network performance makes me question the novelty of the paper. The author should explain their contributions in more detail.
2. The proposed fusion strategy includes MLP, Attention and Convolution. The results in Table 3 reveals the superiority of using MLP. However, the author did not analyze why MLP outperforms the other two fusion methods, but merely compared them in terms of performance. I suggest that a more in-depth discussion would help reviewers better understand your method.
3. The paper claims that "By incorporating video, keypoints and optical flow, our model can effectively capture movement patterns and hand gestures ". However, I cannot find any experiment or visualization results to support this point.
4. The paper claims the distillation technique could maintain the high performance while reducing the computational cost and resource requirements. I believe that a comparison of training time, parameter count, and data preprocessing time between this method and other methods should be provided to support this point. From my viewpoint, keypoint extraction and optical flow computation consume much time.
5. The proposed sign pyramid network is similar to the architecture in the TwoStream-SLR [chen et al., 2022b]. I suggest the author rigorously explains the differences or cites this paper in the proper position.