Out of Sight: A Framework for Egocentric Active Speaker Detection

This paper introduces a transformer-based approach, Out of Sight, that can model all the speaking activities from three prediction targets (visible speakers, camera wearer, and global background speech). The proposed method consists of 3 building blocks: Encoder, Multimodal Mixer, and Decoder. The encoder is a series of convolutional networks that embeds the visual and audio features. The multimodal mixer performs cross-modal attention and aggregates relevant information from audio-visual modalities. The decoder maps all types of tokens (audio, visual, and camera wearer) into a common representation space and predicts the final prediction. To further improve the performance, it uses a technique of long-term feature modeling by incorporating an extended temporal sampling.

The authors propose a method for active speaker detection of egocentric videos. Unlike existing works that focus on YouTube or broadcast videos, ASD for egocentric videos brings additional challenges such as head movement and camera wearer's speech. The paper proposes a 3-stage architecture consisting of (1) modality encoders, (2) multimodal mixer, (3) speech decoder. In particular, a learnable token helps to model speech from the camera wearer. Short- and long-term architectures enable effective multi-modal fusion and extended temporal modelling. The authors use additional tricks like weighted visual loss and position of face in order to improve performance. The method is evaluated on the recent Ego4D dataset which contains various egocentric videos including human speech.

Paper proposed a novel approach for the Active Speaker Detection (ASD) problem in egocentric data (esp. First Person Video (FSD)). This problem is relatively unexplored. The challenges for ASD in FPV is mainly to the "invisibility" of the camera wearer in the video which the SOTA ASD algorithms cannot handle correctly.

The proposed method uses multimodality to overcome this challenge via 3 building blocks: (i) Modality Encoder; (ii) Mutlimodal Mixer; (iii) Speech Decoder.

Experiments were performed to compare against SOTA ASD in FPV methods for the Ego4D dataset.

This paper proposes O2S, a framework for egocentric active speaker detection. Most active speaker detection literatures fall in commercial movies and social media videos, while egocentric videos are less investigated. O2S consists of three stages: 1) An audio encoder and a video encoder are employed to obtain audio features and visual face features. 2) A transformer serves as multimodal mixer to aggregate information from audio and video. 3) Another transformer serves as speech decoder to predict speech event for each face feature, audio feature, and an additional feature for the invisible camerawearer. There are some additional changes made for egocentric videos. First, face positions are added in the visual feature. Second, as egocentric videos may present many blurred faces due to fast motion, noisy faces are less contributed to the loss. Experiments are conducted on the Ego4D dataset for two tasks: Active Speaker Detection of visible targets (vASD) and egocentric Active Speaker Detection (eASD).

We emphasize that the main difference in O2S is the joint prediction of the egocentric active speaker detection (egoASD) and the standar active speaker prediction. EgoASD is a unique element that emerges in the egocentric domain. As outlined in the paper, previous ASD approaches regard out-of-screen speech events as noise and work towards inhibiting the network response in these scenarios. We are the the first method to fully make sense of such events integrating the seamlessly into the egocentric ASD framework

To the best of our knowledge, all current methods try to model this prediction (egoASD) exclusively from the audio stream. O2S is the only method that can jointly model both prediction tasks, while showing a significant improvement in both metrics.

Given its simplicity (audio encoder, video encoder, and transformer head) we see O2S as a starting point to direct the attention of the community towards the detection of speech events in the egocentric domain.

In addition to supporting the selected metric, we kindly remind Reviewers (in particular 1nZN) that papers published within 4 months before the submission are considered concurrent work according to the conference rules [R5]. Therefore no paper should be penalized for not comparing with them. This is the case of the ArXiv report STHG originally released on 18 June of 2023.

[R5] https://iclr.cc/Conferences/2024/ReviewerGuide

Moreover, since both SPELL [R6] and STHG [R7] are 3-page non-peer review reports submitted to ArXiv, the conference rules also state that we are not required to compare with them, and definitely not to be penalized for the performance comparison.

[R6] Kyle Min, Intel Labs at Ego4D Challenge 2022: A Better Baseline for Audio-Visual Diarization, arxiv

[R7] Kyle Min, STHG: Spatial-Temporal Heterogeneous Graph Learning for Advanced Audio-Visual Diarization

In addition to the individual replies to each reviewer, we would like to clarify our use of the mAP metric in the paper as a response of general interest. To the authors’ best understanding, the evaluation metric for the Ego4D validation set is effectively mAP, even though mAP@0.5 is written in the heading of Table 21 in the supplementary material of the Ego4D paper. We support our claim with the following points.

First, we kindly refer the reviewers and AC to the code released by Eric Zhongcong Xu (Ego4D author in the ASD task) which can be found in [R1]. There should be no confusion with the other Ego4D repository for the audiovisual task located at [R2], since that second repository simply redirects to Xu’s implementation of the TalkNet Ego4D baseline, the redirection can be found in [R3].

[R1] https://github.com/zcxu-eric/Ego4d_TalkNet_ASD

[R2] https://github.com/EGO4D/audio-visual

[R3] https://github.com/EGO4D/audio-visual/blob/main/active-speaker-detection/active_speaker/TalkNet_ASD/check_out_talknet.sh

Xu’s code repository shows that inference in the validation set is performed using the face crops provided by the ego4d annotation file, we refer the reviewers and AC to the code snippets at [R4] and [R5]. We also refer the reviewers to the issue #5 [R6] in the same repository, which also clarifies that the results are to be reproduced with the ground-truth face crops.

[R4] https://github.com/zcxu-eric/Ego4d_TalkNet_ASD/blob/ab9f345efc49fd70ed163c6cca674c3aff88e2b6/dataLoader.py#L230

[R5] https://github.com/zcxu-eric/Ego4d_TalkNet_ASD/blob/ab9f345efc49fd70ed163c6cca674c3aff88e2b6/dataLoader.py#L68

[R6] https://github.com/zcxu-eric/Ego4d_TalkNet_ASD/issues/5

Second, in the same thread, it is also asserted that the ground-truth face-crops can be used for inference in validation. And the tracked face crops are reserved for the test set (available only for the challenge). This means that only the results on the test-set for the Ego4D challenge are built from a tracker output. In such a scenario, the metric mAP@0.X makes sense, as the spatial accuracy of the detection must be taken into account.

Third, in the original versions of the two arxiv reports SPELL [R4] and STHG [R5] that Reviewer 1nZN uses to support his/her claims, “mAP” is the metric used to report the performance of SPELL and STHG (Tables 3 and Table 1 respectively). We further note that the TalkNet results of Ego4D are also included in the mentioned tables using “mAP”, just like in our main result table.

[R4] https://arxiv.org/pdf/2210.07764v1.pdf [R5] https://arxiv.org/pdf/2306.10608v1.pdf

In a similar fashion, the paper by Wang et al. “LoCoNet” (whose authors overlap with the authors of Ego4D) also uses the mAP evaluation metric in the validation set. Once again the performance of Ego4D talk-net baseline is reported as mAP in the very same table.

To complement our shared remarks, we would like to raise the attention of the reviewers and AC to the fact that SPELL [R6] and STHG [R7] ArXiv reports were modified just before the release of the reviews (SPELL on October 29 and STHG on October 31). These new revisions introduced minor changes, but critically included the dual evaluation metric (mAP and mAP@0.5) in tables 3 and 1, with a much higher mAP (up to 12% more) than the original version.

[R6] Kyle Min, Intel Labs at Ego4D Challenge 2022: A Better Baseline for Audio-Visual Diarization, arxiv

[R7] Kyle Min, STHG: Spatial-Temporal Heterogeneous Graph Learning for Advanced Audio-Visual Diarization

While we welcome (and share) reviewers’ 1nZN enthusiasm to find the most recent literature in the ASD subject, to the point where he/she managed to spot two new ArXiv revisions submitted just 72 and 24 hours before the end of ICLR review period. We must be emphatic on the fact that the reviewers' claim for lack of state-of-the-art performance are substantiated by revisions made on the 28 of October and 31 of October. How could the authors be aware, compare, or discuss such results if their paper was submitted 1 month before (September 28) any of these ArXiv revisions were uploaded?.

We have searched for code releases of SPELL (Min et al.) and STHG (Min). An official release by the same author named “GraVi-T” [R8] was also updated on October 29, However, we could not make a more in-depth exploration, since the commit of that day did not provide a functioning code base to train, or at least to perform inference on Ego4D. It only contained a copy of the results in the newest version of [R6], [R7] and reflected on Reviewer 1nZN’s comments.
[R8] https://github.com/IntelLabs/GraVi-T

We conclude by re-stating that, to the best of our knowledge, the metrics and evaluation procedure are a direct and fair comparison to those in Ego4D validation set. We are eager to compare and include related works, but it is beyond the authors’ capability to benchmark and discuss results which are updated in ArXiv 1 month after the paper submission.

This paper received mixed but overall negative ratings (1, 3, 6, 6). Two reviewers who were in favor (kwFT, dygL) did not provide detailed explanations for why the paper should be accepted, while the other two (2WYw, 1nZN) provided much more detailed and convincing arguments for rejection. There were several concerns about novelty, weak empirical evidence due to marginal improvements, and some doubts about experimental settings and evaluation metrics. The rebuttal addressed some of the concerns, but during the author discussion period, one critical issue stood out: the evaluation metric. The authors claimed that mAP should be the correct metric, while one of the reviewers asserted that mAP@0.5 should be used instead. The reviewer provided detailed justifications for why this is the case, citing the evaluation script provided by the official Ego4D challenge organizers. The authors did the same, citing a repository from one of the authors of the Ego4D paper. To verify the correctness of the claim, this meta-reviewer has contacted the Ego4D ASD challenge organizers, and they confirmed that mAP@0.5 is indeed the correct metric for the official challenge since a face detector is involved. Given the inaccuracies involved in the paper, we are recommending rejection at this time in the hope that the authors will adopt the correct evaluation metric.

Reject