| Paper ID | SPE-20.6 |
| Paper Title |
A MULTI-VIEW APPROACH TO AUDIO-VISUAL SPEAKER VERIFICATION |
| Authors |
Leda Sari, University of Illinois at Urbana-Champaign, United States; Kritika Singh, Jiatong Zhou, Lorenzo Torresani, Nayan Singhal, Yatharth Saraf, Facebook AI, United States |
| Session | SPE-20: Speaker Recognition 4: Applications |
| Location | Gather.Town |
| Session Time: | Wednesday, 09 June, 14:00 - 14:45 |
| Presentation Time: | Wednesday, 09 June, 14:00 - 14:45 |
| Presentation |
Poster
|
| Topic |
Speech Processing: [SPE-SPKR] Speaker Recognition and Characterization |
| IEEE Xplore Open Preview |
Click here to view in IEEE Xplore |
| Virtual Presentation |
Click here to watch in the Virtual Conference |
| Abstract |
Although speaker verification has conventionally been an audio-only task, some practical applications provide both audio and visual streams of input. In these cases, the visual stream provides complementary information and can often be leveraged in conjunction with the acoustics of speech to improve verification performance. In this study, we explore audio-visual approaches to speaker verification, starting with standard fusion techniques to learn joint audio-visual (AV) embeddings, and then propose a novel approach to handle cross-modal verification at test time. Specifically, we investigate unimodal and concatenation based AV fusion and report the lowest AV equal error rate (EER) of 0.7% on the VoxCeleb1 dataset using our best system. As these methods lack the ability to do cross-modal verification, we introduce a multi-view model which uses a shared classifier to map audio and video into the same space. This new approach achieves 28% EER on VoxCeleb1 in the challenging testing condition of cross-modal verification. |
