EEG-fNIRS based hybrid Image reconstruction and classification for BCI

Abstract

The ever-increasing human-machine interaction and advancement in socio-technical systems have made it essential to analyze the vital human factors such as mental workload, vigilance, fatigue, and stress, etc via monitoring brain states. Similarly, brain signals are becoming paramount for rehabilitation and assistive purposes in fields such as brain-computer interface (BCI), closed-loop neuromodulation for neurological disorders, etc. The complex, non-stationary, and very low signal-to-noise ratio of brain signals poses a significant challenge for researchers to design robust and reliable BCI systems outside the laboratory environment. In this work, I have presented a novel recurrence plots (RPs) based time-distributed convolutional neural network and long short term memory (CNN-LSTM) algorithm for four class functional near-infrared spectroscopy (fNIRS) BCI, electroencphelography (EEG) BCI and Hybrid EEG-fNIRS BCI. The acquired brain signals are projected into a non-linear dimension with RPs and fed into the CNN which extracts the important features and then LSTM learns the chronological and time-dependent relations. The average accuracy achieved with the proposed model is 79.7% with fNIRS 83.6% with EEG and 88.5%. for hybrid EEG-Fnirs BCI. While the maximum accuracies achieved are 85.9%, 88.1% and 92.4%, respectively. The results confirm the viability of RPs based deep learning algorithm for successful BCI systems.