Facial expression recognition in-the-wild using blended feature attention network

Abstract

Facial expression (FE) analysis plays a crucial role in various fields, such as affective computing, marketing, and clinical evaluation. Despite numerous advances, research on FE recognition (FER) has recently been proceeding from confined laboratory circumstances to in-the-wild environments. FER is still an arduous and demanding problem due to occlusion and pose changes, intraclass and intensity variations caused by illumination, and insufficient training data. Most state-of-the-art (SOTA) approaches use the entire face for FER. However, past studies on psychology and physiology reveal that the mouth and eyes reflect the variations of various FEs, which are closely related to the manifestation of emotion. A novel method is proposed in this study to address some of the issues mentioned above. First, modified homomorphic filtering (MHF) is employed to normalize the illumination, then the normalized face image is cropped into five local regions to emphasize expression-specific characteristics. Finally, a unique blended feature attention network (BFAN) is designed for FER. BFAN consists of both residual dilated multiscale (RDMS) feature extraction modules and spatial and channel-wise attention (CWA) modules. These modules help to extract the most relevant and discriminative features from the high-level (HL) and low-level (LL) features. Then, both feature maps are integrated and passed on to the dense layers followed by a softmax layer to compute probability scores. Finally, the Choquet fuzzy integral is applied to the computed probability scores to get the final outcome. The superiority of the proposed method is exemplified by comparing it with 18 existing approaches on seven benchmark datasets.