| Paper ID | ASPS-5.4 |
| Paper Title |
How to Use Time Information Effectively? Combining with Time Shift Module for Lipreading |
| Authors |
Mingfeng Hao, Mutallip Mamut, Nurbiya Yadikar, Alimjan Aysa, Kurban Ubul, Xinjiang University, China |
| Session | ASPS-5: Audio & Images |
| Location | Gather.Town |
| Session Time: | Thursday, 10 June, 16:30 - 17:15 |
| Presentation Time: | Thursday, 10 June, 16:30 - 17:15 |
| Presentation |
Poster
|
| Topic |
Applied Signal Processing Systems: Signal Processing Systems [DIS-EMSA] |
| IEEE Xplore Open Preview |
Click here to view in IEEE Xplore |
| Virtual Presentation |
Click here to watch in the Virtual Conference |
| Abstract |
In this paper, we improve this point. Firstly, Time Shift Module (TSM) is applied to two different front-ends (full 2D CNN based and mixture of 2D and 3D convolution) to enhance the ability of time information extraction. Secondly, we verify the influence of different shift proportion of TSM and different frame rate input on extracting time information. Thirdly, we compare the number of moving frames in TSM. We find that shifting in the time dimension by ±1 can better extract the short-term time features, and excessive movement will reduce the performance of 2D CNN. We have verified our method on two challenging word-level lipreading datasets LRW and LRW-1000, and achieved new state-of-the-art performance. |
