Identifying humanitarian information for emergency response by modeling the correlation and independence between text and images |
| |
| Institution: | 1. Center for Studies of Information Resources, Wuhan University, Bayi Rd. 299, Wuhan, Hubei 430072, China;2. School of Information Management, Wuhan University, Bayi Rd 299, Wuhan 430072, China;1. Guangxi Key Lab of Multi-source Information Mining and Security, Guangxi Normal University, Guilin, Guangxi, China;2. College of Cyber security, Jinan University, Guangzhou China;3. The College of Information Science and Engineering, Guilin University of Technology, Guilin, China;1. School of Management and Economics, University of Electronic Science and Technology of China, No.2006, Xiyuan Ave, West Hi-Tech Zone, 611731 Chengdu, Sichuan, PR China;2. The Walker School of Business and Technology, Webster University, 470 E Lockwood Ave, Webster Groves, MO 63119, United States;3. The James F. Dicke College of Business Administration, Ohio Northern University, 525 S Main St, Ada, OH, United States |
| |
| Abstract: | Information residing in multiple modalities (e.g., text, image) of social media posts can jointly provide more comprehensive and clearer insights into an ongoing emergency. To identify information valuable for humanitarian aid from noisy multimodal data, we first clarify the categories of humanitarian information, and define a multi-label multimodal humanitarian information identification task, which can adapt to the label inconsistency issue caused by modality independence while maintaining the correlation between modalities. We proposed a Multimodal Humanitarian Information Identification Model that simultaneously captures the Correlation and Independence between modalities (CIMHIM). A tailor-made dataset containing 4,383 annotated text-image pairs was built to evaluate the effectiveness of our model. The experimental results show that CIMHIM outperforms both unimodal and multimodal baseline methods by at least 0.019 in macro-F1 and 0.022 in accuracy. The combination of OCR text, object-level features, and the decision rule based on label correlations enhances the overall performance of CIMHIM. Additional experiments on a similar dataset (CrisisMMD) also demonstrate the robustness of CIMHIM. The task, model, and dataset proposed in this study contribute to the practice of leveraging multimodal social media resources to support effective emergency response. |
| |
| Keywords: | |
| 本文献已被 ScienceDirect 等数据库收录！ |
|
