Visual Decoding of Brain from fMRI using Machine Learning

Abstract

Reconstructing fine features of natural images from functional Magnetic Resonance Imaging (fMRI) of never-before-seen data is challenging. Acquiring a large-scale fMRI from a subject viewing natural images is a highly resource-intensive endeavor. Due to limited sample data, the recent approaches use a pre-trained generative network or unsupervised learning method for extracting features of reconstructing natural images during network training. However, these reconstructed images are not clear and unidentifiable unless their ground truths are available. Further, these results cannot be enhanced up to identifiable natural images because the training model will hardly learn to construct coarse features of never-before-seen data. However, it was observed from the state-ofthe-art results that the reconstructed natural images exhibit consistent structure, shape, and coarse features across subjects although each subject’s fMRI is different. Leveraging nearly uniform reconstructed test images across subjects, we reorient our research focus. The proposed approach utilizes Beliy’s model to reconstruct the images from the test subject’s fMRI followed by the proposed denoising model, based on the pre-trained classification model and generative adversarial Network BigGAN, to produce identifiable images. Our proposed approach using our own denoising model gives up to 70% identification accuracy in comparison with the 16%, and 30% identification accuracy given by the adversarial autoencoder and variational autoencoder, respectively.