A Multimodal Deep Learning Approach for Depression Detection using Audio and Video Data

T. Srajan Kumar; Kandula Siri Chandana; Gavara Archana; Gundeti Dhanush Reddy; Jonna Madhu Reddy

doi:10.55524/ijirem.2026.13.2.1

Abstract

Depression is a significant mental health problem in the global society and tends to be undiagnosed because there are no convenient and effective screening procedures. Recognizing depressive symptoms early is important to be able to intervene and treat in time. In this study, the authors suggest a multimodal deep learning-based system of depression detection that takes audio and video inputs to detect patterns of depression. The system evaluates speech features and facial behavioural indicators obtained on the foundation of documented discussions. OpenFace extracts visual (facial action units, gaze direction, and head pose) and speech (processes to extract acoustic aspects of interest in terms of expressing emotions) signals. The model applied in deep learning is the Convolutional Neural Network (CNN) to learn discriminative representations of the extracted features. Audio and visual modalities predicted are then joined together through a fusion mechanism to enhance the overall effectiveness of the classification. The findings of the experiments prove that multimodal analysis is more accurate at detecting depression than unimodal one. The suggested system offers a non-intrusive and scalable system, which can aid mental health practitioners with the initial screening of depression.

Keywords

Depression Detection, Multimodal Learning, Deep Learning, Convolutional Neural Networks, Audio Analysis, facial Behaviour Analysis

References

J. Gratch, R. Artstein, G. M. Lucas, et al., “The Distress Analysis Interview Corpus of Human and Computer Interviews,” in Proc. Int. Conf. Language Resources and Evaluation (LREC), 2014, pp. 3123–3128. Available from: https://tinyurl.com/343abt3c
Pampouchidou, O. Simantiraki, C. Vazakopoulou, et al., “Automatic Assessment of Depression Based on Visual Cues: A Systematic Review,” IEEE Trans. Affective Computing, vol. 10, no. 4, pp. 445–470, 2019. Available from: https://doi.org/10.1109/TAFFC.2017.2724035
Z. Yang, X. Xia, and L. Zhao, “Multimodal Depression Detection Using Audio, Visual and Textual Features,” IEEE Access, vol. 8, pp. 194898–194907, 2020.
S. Al Hanai, M. Ghassemi, and J. Glass, “Detecting Depression with Audio/Text Sequence Modeling of Interviews,” in Proc. Interspeech, 2018, pp. 1716–1720. Available from: https://www.isca-archive.org/interspeech_2018/alhanai18_interspeech.pdf
F. Eyben, K. R. Scherer, B. W. Schuller, et al., “The Geneva Minimalistic Acoustic Parameter Set (GeMAPS) for Voice Research and Affective Computing,” IEEE Trans. Affective Computing, vol. 7, no. 2, pp. 190–202, 2016. Available from: https://doi.org/10.1109/TAFFC.2015.2457417
T. Baltrusaitis, P. Robinson, and L. P. Morency, “OpenFace: An Open-Source Facial Behavior Analysis Toolkit,” in Proc. IEEE Winter Conf. Applications of Computer Vision (WACV), 2016, pp. 1–10. Available from: https://doi.org/10.1109/WACV.2016.7477553
P. Tzirakis, G. Trigeorgis, M. A. Nicolaou, B. W. Schuller, and S. Zafeiriou, “End-to-End Multimodal Emotion Recognition Using Deep Neural Networks,” IEEE J. Selected Topics Signal Processing, vol. 11, no. 8, pp. 1301–1309, 2017. Available from: https://doi.org/10.1109/JSTSP.2017.2764438
S. Dham, A. Gupta, and R. Bansal, “Depression Detection Using Machine Learning Techniques on Audio and Visual Data,” Procedia Computer Science, vol. 167, pp. 2313–2321, 2020.
Y. Yang, C. Fairbairn, and J. F. Cohn, “Detecting Depression Severity from Vocal Prosody,” IEEE Trans. Affective Computing, vol. 4, no. 2, pp. 142–150, 2013. Available from: https://doi.org/10.1109/T-AFFC.2012.38
H. Dibeklioglu, Z. Hammal, and J. F. Cohn, “Dynamic Multimodal Measurement of Depression Severity Using Facial Expressions and Head Movements,” IEEE J. Biomedical and Health Informatics, vol. 22, no. 2, pp. 526–536, 2018. Available from: https://doi.org/10.1109/JBHI.2017.2676878
S. Cummins, M. Scherer, J. Krajewski, et al., “A Review of Depression and Suicide Risk Assessment Using Speech Analysis,” Speech Communication, vol. 71, pp. 10–49, 2015. Available from: https://doi.org/10.1016/j.specom.2015.03.004
B. W. Schuller, S. Steidl, A. Batliner, et al., “The INTERSPEECH 2013 Computational Paralinguistics Challenge: Social Signals, Conflict, Emotion, Autism,” in Proc. Interspeech, 2013, pp. 148–152. Available from: https://mediatum.ub.tum.de/doc/1189705/document.pdf
Gideon, S. Khorram, Z. Aldeneh, et al., “Progressive Neural Networks for Transfer Learning in Emotion Recognition,” in Proc. ACM Multimedia Conf., 2017, pp. 1097–1105. Available from: https://arxiv.org/abs/1706.03256
M. Valstar, J. F. Cohn, T. Baltrusaitis, et al., “AVEC 2016 – Depression, Mood, and Emotion Recognition Workshop and Challenge,” in Proc. ACM Int. Conf. Multimedia, 2016, pp. 3–10. Available from: https://dl.acm.org/doi/abs/10.1145/2988257.2988258
S. SJ, E. A. Suvi, R. J. Jajin, A. Jagan, and S. PS, “Sign language recognition using deep learning: A systematic review of models and approaches,” International Journal of Innovative Research in Computer Science and Technology (IJIRCST), vol. 14, no. 1, pp. 34–42, 2026 Available from: https://doi.org/10.55524/ijircst.2026.14.1.5
G. Tiwary, S. Chauhan, and K. K. Goyal, “Multimodal depression detection using audio visual cues,” in 2023 International Conference on Computer Science and Emerging Technologies (CSET), Bangalore, India, 2023, pp. 1–5, Available from: https://doi.org/10.1109/CSET58993.2023.10346770

Cites this article as

T. S. Kumar, K. S. Chandana, G. Archana, G. D. Reddy, J. M. Reddy, "A Multimodal Deep Learning Approach for Depression Detection using Audio and Video Data", International Journal of Innovative Research in Engineering and Management (IJIREM), Vol-13, Issue-2, Page No-1-6, 2026. Available from: https://doi.org/10.55524/ijirem.2026.13.2.1