How Do You Perceive My Face? Recognizing Facial Expressions in Multi-Modal Context by Modeling Mental Representations - Apprentissage de modèles visuels à partir de données massives
Communication Dans Un Congrès Année : 2024

How Do You Perceive My Face? Recognizing Facial Expressions in Multi-Modal Context by Modeling Mental Representations

Comment Perçois-tu Mon Visage ? Reconnaissance des Expressions Faciales dans un Contexte Multi-Modal par la Modélisation des Représentations Mentales

Résumé

Facial expression perception in humans inherently relies on prior knowledge and contextual cues, contributing to efficient and flexible processing. For instance, multi-modal emotional context (such as voice color, affective text, body pose, etc.) can prompt people to perceive emotional expressions in objectively neutral faces. Drawing inspiration from this, we introduce a novel approach for facial expression classification that goes beyond simple classification tasks. Our model accurately classifies a perceived face and synthesizes the corresponding mental representation perceived by a human when observing a face in context. With this, our model offers visual insights into its internal decision-making process. We achieve this by learning two independent representations of content and context using a VAE-GAN architecture. Subsequently, we propose a novel attention mechanism for context-dependent feature adaptation. The adapted representation is used for classification and to generate a context-augmented expression. We evaluate synthesized expressions in a human study, showing that our model effectively produces approximations of human mental representations. We achieve State-of-the-Art classification accuracies of 81.01% on the RAVDESS dataset and 79.34% on the MEAD dataset. We make our code publicly available 5 .
Fichier principal
Vignette du fichier
2409.02566v1.pdf (22.64 Mo) Télécharger le fichier
Origine Fichiers produits par l'(les) auteur(s)

Dates et versions

hal-04778736 , version 1 (12-11-2024)

Licence

Identifiants

  • HAL Id : hal-04778736 , version 1

Citer

Florian Blume, Runfeng Qu, Pia Bideau, Martin Maier, Rasha Abdel Rahman, et al.. How Do You Perceive My Face? Recognizing Facial Expressions in Multi-Modal Context by Modeling Mental Representations. GCPR 2024 - German Conference on Pattern Recognition, Sep 2024, Munich, Germany. pp.1-28. ⟨hal-04778736⟩
5 Consultations
7 Téléchargements

Partager

More