Deep Learning based Emotion Charting for Healthy and Cognitively Impaired Subjects using Physiological Signals

Abstract

Parkinson’s Disease(PD) is the second most common neuro degenerative disorder, resulting in cognitive impairments in emotion recognition. The deficit of emotional expression poses challenges to the health care services provided and the quality of life of PD patients. Emotion charting for cognitively impaired patients is challenging compared to healthy subjects. The continuous monitoring of cognitively impaired patients with physiological signals such as Electroencephalogram(EEG),Electro- cardiograms (ECG),and Galvanicsk in response(GSR) provide physiological health for the patients. Novel research trends in corporate the sephysiological signals by reflecting actual and in trinsice motional states resulting in more reliable, natural, and meaningful human-computer interaction with applications in entertainment consumption behavior, interactive brain-computer interface, and monitoring of the psychological health of patients. Young adults and children commonly use technology for human computer interaction and entertainment consumption behavior. The main challenges in this domain are the low emotion recognition performance for PD patients duetoloss of dopam in ergicneurons, low performance for memory-induced emotions due to weaker signal and concentration loss, and lack of data set of children. The previous research lacked to directly explores the one-dimensional convolutional recurrent neural network deep learning model, suitable for long, continuous, and repetitive patterns of EEG, ECG, and GSR for the emotion charting of cognitively impaired patients and memory-induced emotion recognition. Other challenges in real-world applications include are duced performance with the increased number of emotion classes, wearable acquisition sensors, and experimental settings such as age group and emotional stimuli provide to the subjects. Similarly, despite the efficacy of 1D-CRNNandELM for physiological signals data, the combination of the set wo is note xplored in the literature. This thesis addresses the sechallengesbyproposinganovel1D-CRNN-ELMarchitec- ture, whichcombinesaone-dimensionalConvolutionalRecurrentNeuralNetwork (1D-CRNN) withanExtremeLearningMachine(ELM),robustfortheemotion detection ofPDpatients,alsoavailableforcrossdatasetlearningwithvariousemo- tions andexperimentalsettings.Intheproposedframework,thepreprocessingof physiologicalsignalsinvolvesbaselineremoval,passbandfiltering,andZ-scorenor- malization. Afterpreprocessing,1D-CRNNarchitecturewiththree1D-CNNlayers (16 filterswiththesizeof1x8each),followedbyanLSTMlayertrainedwithpre- processedphysiologicalsignals.Thetrained1D-CRNNarchitectureisusedasthe feature extractorofphysiologicalsignals.Theextracteddeepfeaturesarethen passed throughanextremelearningmachineclassifiertoclassifyemotionsbothin a categorical(fear,happy,sad,disgust,anger,andsurprise)anddimensionalmodel (four quadrantsofhighvalencehigharousal(HVHA),highvalencelowarousal (HVLA), lowvalencehigharousal(LVHA),andlowvalencelowarousal(LVLA)). This researchalsoexploredfine-tuningforcross-datasetlearningofemotionsamong Parkinson’sdiseasepatientsdatasetandpubliclyavailabledatasetsofhealthysub- jects. This researchcontributedanovel,robustandgenericframeworktohandlehealthy i and cognitivelyimpairedpatientsforemotionrecognition.Theproposedframework outperformstherecognitionperformanceofexistingtechniqueswithpubliclyavail- able datasetsofAMIGOS,DREAMER,andSEED-IV,withthePDpatientsdataset, and providesbenchmarkbaselineresultsformemory-inducedandchildrendatasets. It improvestherecognitionperformancecomparedtothestate-of-the-artforboth categorical anddimensionalmodelsofemotionchartingsubject-independentstudy with wirelesssensorsissuitableforless-constrainedreal-worldenvironments.Italso incorporatedthelessexploredECGandGSRsignalsforlessinvasive,low-cost,wear- able emotionrecognitionwithmultimodalfusionatthedecisionlevel.Thisresearch providesanoriginalattempttoexplorethedeeplearningmodelforPDpatientsand a noveldatasetofself-inducedemotionswithautobiographicalmemoriesevokedby the relevantwords.Theinducedemotionsthroughexternalstimuliareoftenmore substantialthantheemotionalexperiencesfeltbyhumansindailyroutines.The self-induced emotionsthroughmemoryrecallsarethemostcommonexperiencesin real-worldscenariosforcontinuousemotionchartingwithanoveldatasetofevoked memory recallstitledMEMO.Thisresearchdevelopedandprovidedanovel,multi- modaldatasetofchildrenandyoungadults(YAAD)withbenchmarkresultspublicly availabletotheresearchcommunityforemotionchartingwithphysiologicalsignals. The proposedmethodoutperformsstate-of-the-artstudiestoclassifyemotionswith publicly availabledatasets,providecross-datasetlearning,andvalidatetherobust- ness ofthedeeplearningframeworkforreal-worldscenariossuchasevokedmemory recalls andpsychologicalhealthcaremonitoringofParkinson’spatients.