| Paper ID | MMSP-7.6 |
| Paper Title |
SEMI-SUPERVISED MULTIMODAL IMAGE TRANSLATION FOR MISSING MODALITY IMPUTATION |
| Authors |
Wangbin Sun, Fei Ma, Yang Li, Shao-Lun Huang, Shiguang Ni, Lin Zhang, Tsinghua University, China |
| Session | MMSP-7: Multimodal Perception, Integration and Multisensory Fusion |
| Location | Gather.Town |
| Session Time: | Friday, 11 June, 13:00 - 13:45 |
| Presentation Time: | Friday, 11 June, 13:00 - 13:45 |
| Presentation |
Poster
|
| Topic |
Multimedia Signal Processing: Signal Processing for Multimedia Applications |
| IEEE Xplore Open Preview |
Click here to view in IEEE Xplore |
| Virtual Presentation |
Click here to watch in the Virtual Conference |
| Abstract |
Missing data is a common problem in multimodal and multi-view learning. It raises a critical challenge for most multimodal algorithms, which are unable to deal with incomplete datasets. Rather than discarding entries with missing modalities, this paper aims to reconstruct the complete image-based multimodal data by imputing missing modalities. We solve the imputation problem as an image translation task, which transforms images in one domain to other domains. Existing image translation techniques either can not fully utilize the information contained in partially complete entries or are limited to the bimodal situation. We propose a semi-supervised algorithm for multimodal learning with missing data, namely Cyclic Autoencoder (CycAE). Specifically, a novel cyclical structure, as well as the correlation among modalities, is integrated to leverage information from complete entries to incomplete ones. Experiments on two multimodal datasets show that our model outperforms state-of-the-art models. Downstream tasks can also benefit from the completed datasets. |
