Javascript must be enabled to continue!
From Bias to Balance: Detecting Facial Expression Recognition Biases in Large Multimodal Foundation Models

Facial expression recognition (FER) systems are increasingly used in human-centered applications, from detecting pain in patients who face difficulties with communication to monitoring physical and emotional well-being. Yet demographic biases in these systems may produce inequitable outcomes across racial groups, particularly when deployed in sensitive settings such as healthcare. While prior research has examined bias in traditional FER architectures, less is known about such disparities in Large Multimodal Foundation Models (LMFMs). We benchmark four leading models—GPT-4o, PaliGemma, Gemini, and CLIP—across three facial-expression datasets containing subjects from multiple racial demographics. We evaluate performance across race and emotion categories and conduct two-proportion z-tests to identify statistically significant differences in misclassification rates. A linear classifier trained on CLIP embeddings achieves accuracies of 95.9% on RADIATE, 90.3% on Tarr, and 99.5% on Chicago Face, outperforming the evaluated zero-shot models. However, high aggregate accuracy does not eliminate demographic disparities. We identify statistically significant differences in misclassification rates between several racial groups and find that, for zero-shot GPT-4o on RADIATE, Anger is misclassified as Disgust 2.1 times more often for Black females than White females. These findings demonstrate that aggregate performance metrics can obscure meaningful demographic differences and highlight the importance of demographic evaluation when developing responsible multimodal AI for human-centered and healthcare applications.

Attachment language: English File type: PDF document Presentation Poster
Updated: 04-11-2026 19:11 - Size: 4.27 MB
Attachment language: English File type: PDF document Presentation File
Updated: 04-11-2026 19:12 - Size: 1.17 MB
Speakers
Vedant HathaliaKaylee ChhuaZhoujinyi WenKevin ZhuSean O'Brien

Back

ACM GreekCHI-CYCHI Symposium 2026


We are excited to invite you to the ACM GreekCHI-CYCHI Symposium 2026, held on November 6-7, 2026 jointly in Athens, Thessaloniki, Nicosia and Online.