Recent efforts on multimodal sentiment analysis (MSA) leverage data from multiple modalities, among which the text modality is heavily relied on. However, the text modality often contains false correlations between text tokens and sentiment labels, leading to errors in sentiment analysis. To address this issue, we propose a new framework, PriSA, which incorporates the preferential fusion and distance-aware contrastive learning. Specifically, we first propose a preferential inter-modal fusion method, which utilizes the text modality to guide the calculation of the inter-modal correlations. Then the resulting inter-modal features are further used to calculate mixed-modal correlations through our proposed distance-aware contrastive learning, which leverages the distance information of the sentiment labels. At last, we identify the sentiment information based on both the mixed-modal correlations and the discriminative intra …
本文针对多模态情感分析中文本模态与情感标签之间存在的虚假关联问题，提出了一种名为PriSA的新框架。该框架首先通过优先跨模态融合方法，利用文本模态引导计算跨模态相关性；随后引入距离感知对比学习，利用情感标签之间的距离信息进一步计算混合模态相关性；最终基于混合模态相关性和判别性类内特征识别情感信息。实验表明该方法能有效缓解文本虚假关联带来的影响，提升多模态情感分析性能。