| Paper ID | IVMSP-27.4 |
| Paper Title |
GENERATING NATURAL QUESTIONS FROM IMAGES FOR MULTIMODAL ASSISTANTS |
| Authors |
Alkesh Patel, Akanksha Bindal, Hadas Kotek, Christopher Klein, Jason Williams, Apple, United States |
| Session | IVMSP-27: Multi-modal Signal Processing |
| Location | Gather.Town |
| Session Time: | Friday, 11 June, 11:30 - 12:15 |
| Presentation Time: | Friday, 11 June, 11:30 - 12:15 |
| Presentation |
Poster
|
| Topic |
Image, Video, and Multidimensional Signal Processing: [IVARS] Image & Video Analysis, Synthesis, and Retrieval |
| IEEE Xplore Open Preview |
Click here to view in IEEE Xplore |
| Virtual Presentation |
Click here to watch in the Virtual Conference |
| Abstract |
Generating natural, diverse, and meaningful questions from images is an essential task for multimodal assistants as it confirms whether they have understood the object and scene in the images properly. The research in visual question answering and visual question generation is a great step. However, this research does not capture questions that a visually-abled person would ask multimodal assistants. Recently published datasets such as KB-VQA, FVQA, and OK-VQA try to collect questions that look for external knowledge which makes them appropriate for multimodal assistants. However, they still contain many obvious and common-sense questions that humans would not usually ask a digital assistant. This paper provides new benchmark dataset that contains questions generated by human annotators keeping in mind what they would ask multimodal digital assistants. Large scale annotations for several hundred thousand images are expensive and time-consuming, so we also present automatic way of generating questions from unseen images. This paper presents an approach for generating diverse and meaningful questions that consider image content and metadata of image. We evaluate our approach using standard evaluation metrics to show the relevance of generated questions with human-provided questions. We also measure the diversity of generated questions using generative strength and inventiveness metrics. |
