| 摘要: |
| 针对多模态虚假新闻识别任务中主流方法难以检测出新闻内容中的事理逻辑错误与常识性错误的问题，提出了一种基于异构知识融合的多模态虚假新闻识别方法。首先引入了事理逻辑抽取以及常识抽取算法，从新闻中抽取出常识以及事理知识，同时根据抽取出的知识从事理图谱和常识图谱中链接到相关的证据。然后设计了一种异构知识融合网络，对知识和新闻进行特征编码，利用异构图注意力网络对知识图进行特征聚合，并通过知识对比模块对图谱中召回的知识与新闻知识进行对比，获得深度对比特征，同时利用协同注意力模块将新闻的图像与文本特征有效融合得到图文融合特征。最后，将对比知识特征与图文融合特征自适应融合，进一步提高模型的识别精度。在Weibo公开数据集和自建数据集的实验结果表明，所提方法的识别准确率分别达到了91.6%和96.9%，与目前主流的方法相比识别性能更好。 |
| 关键词: 多模态虚假新闻识别 异构知识融合 知识图谱 跨模态融合 |
| DOI：10.20079/j.issn.1001-893x.241011004 |
|
| 基金项目:国家自然科学基金面上项目（62176171） |
|
| Multimodal Fake News Detection Based on Heterogeneous Knowledge Fusion |
| LIU Xin,DAI Lican,ZHANG Haiying,WANG Shengze |
| (1.Southwest China Institute of Electronic Technology,Chengdu 610036,China;2.College of Computer Science and Technology,Harbin Engineering University,Harbin 150001,China) |
| Abstract: |
| To address the challenges faced by current models in detecting logical inconsistencies and common-sense errors in multimodal fake news detection tasks,a multimodal fake news detection method based on a heterogeneous knowledge fusion network is proposed.This approach initially introduces algorithms for extracting logical reasoning and common-sense knowledge,enabling the extraction of relevant information from news articles while also identifying pertinent evidence from reasoning and common-sense knowledge graphs according to the extracted knowledge.Subsequently,a heterogeneous knowledge fusion network is designed,which first encodes the features of both knowledge and news.The method employs a heterogeneous graph attention network to aggregate features from the knowledge graph and utilizes a knowledge comparison module to contrast the knowledge obtained from the graph with the news knowledge,thus deriving deep comparative features.A collaborative attention module effectively fuses image and text features to generate multimodal features.Finally,the comparative knowledge features are adaptively fused with the multimodal features,further enhancing the model搒 recognition accuracy.Experimental results on the Weibo public dataset and a self-constructed dataset demonstrate that the proposed method achieves accuracy rates of 91.6% and 96.9% respectively,outperforming current mainstream methods. |
| Key words: multimodal fake news detection heterogeneous knowledge fusion knowledge graph cross modal fusion |
