摘要
在结合脑电(EEG)信号与人脸图像的双模态情感识别领域中,通常存在两个挑战性问题:(1)如何从EEG信号中以端到端方式学习到更具显著性的情感语义特征;(2)如何充分利用双模态信息,捕捉双模态特征中情感语义的一致性与互补性。为此,提出了多层次时空特征自适应集成与特有-共享特征融合的双模态情感识别模型。一方面,为从EEG信号中获得更具显著性的情感语义特征,设计了多层次时空特征自适应集成模块。该模块首先通过双流结构捕捉EEG信号的时空特征,再通过特征相似度加权并集成各层次的特征,最后利用门控机制自适应地学习各层次相对重要的情感特征。另一方面,为挖掘EEG信号与人脸图像之间的情感语义一致性与互补性,设计了特有-共享特征融合模块,通过特有特征的学习和共享特征的学习来联合学习情感语义特征,并结合损失函数实现各模态特有语义信息和模态间共享语义信息的自动提取。在DEAP和MAHNOB-HCI两种数据集上,采用跨实验验证和5折交叉验证两种实验手段验证了提出模型的性能。实验结果表明,该模型取得了具有竞争力的结果,为基于EEG信号与人脸图像的双模态情感识别提供了一种有效的解决方案。
There are usually two challenging issues in the field of bimodal emotion recognition combining ElectroEncephaloGram(EEG)and facial images:(1)How to learn more significant emotionally semantic features from EEG signals in an end-to-end manner;(2)How to effectively integrate bimodal information to capture the coherence and complementarity of emotional semantics among bimodal features.In this paper,a bimodal emotion recognition model is proposed via the adaptive integration of multi-level spatial-temporal features and the fusion of specific-shared features.On the one hand,in order to obtain more significant emotionally semantic features from EEG signals,a module,called adaptive integration of multi-level spatialtemporal features,is designed.The spatial-temporal features of EEG signals are firstly captured with a dualflow structure before the features from each level are integrated by taking into consideration the weights deriving from the similarity of features.Finally,the relatively important feature information from each level is adaptively learned based on the gating mechanism.On the other hand,in order to leverage the emotionally semantic consistency and complementarity between EEG signals and facial images,one module fusing specificshared features is devised.Emotionally semantic features are learned jointly through two branches:specificfeature learning and shared-feature learning.The loss function is also incorporated to automatically extract the specific semantic information for each modality and the shared semantic information among the modalities.On both the DEAP and MAHNOB-HCI datasets,cross-experimental verification and 5-fold cross-validation strategies are used to assess the performance of the proposed model.The experimental results and their analysis demonstrate that the model achieves competitive results,providing an effective solution for bimodal emotion recognition based on EEG signals and facial images.
作者
孙强
陈远
SUN Qiang;CHEN Yuan(Department of Communication Engineering,School of Automation and Information Engineering,Xi’an University of Technology,Xi’an 710048,China;Xi’an Key Laboratory of Wireless Optical Communication and Network Research,Xi’an 710048,China)
出处
《电子与信息学报》
EI
CAS
CSCD
北大核心
2024年第2期574-587,共14页
Journal of Electronics & Information Technology
基金
西安市科技计划项目(22GXFW0086)
西安市碑林区科技计划项目(GX2243)
西安理工大学研究生校企协同创新基金(310/252062108)。
关键词
双模态情感识别
脑电
人脸图像
多层次时空特征
特征融合
Bimodal emotion recognition
ElectroEncephaloGram(EEG)
Facial image
Multi-level spatialtemporal features
Feature fusion
