Analysis of Social Bias in Vision-Language Models via Explainability Methods Görsel-Dil Modellerinde Sosyal Önyarginin Açiklanabilirlik Yöntemleriyle ?Incelenmesi
34th Signal Processing and Communications Applications Conference, SIU 2026, İstanbul, Türkiye, 7 - 10 Temmuz 2026, (Tam Metin Bildiri)
- Yayın Türü: Bildiri / Tam Metin Bildiri
- Doi Numarası: 10.1109/siu71813.2026.11636950
- Basıldığı Şehir: İstanbul
- Basıldığı Ülke: Türkiye
- Anahtar Kelimeler: bias, explainability, social stereotypes, Vision-language models
- Galatasaray Üniversitesi Adresli: Evet
Özet
Despite strong multimodal performance, vision-language models such as CLIP remain insufficiently examined for social bias at the level of visual representation. Using four explainability methods and a pixel-normalized face attention density metric derived from SAM-based semantic segmentation, we systematically investigate CLIP's visual attention distribution across race and gender. Results show that under identical occupational prompts, CLIP consistently allocates greater facial attention to Black individuals compared to White individuals, and that female individuals receive higher facial attention rates in certain occupational scenarios. These patterns suggest a systematic asymmetry in CLIP's visual attention stemming from the model's representational structure rather than coincidence.