CNN-RNN Based Model for Automated Radiology Report Generation

Abstract

The automated generation of radiology reports provided X-rays and has tremendous potential to operationally enhance the clinical diagnosis of diseases for patients. A new landmark of research is being created by using hybrid approaches based on Natural Language Processing and Computer Vision Techniques to create auto medical report generation systems. The auto report generator, producing radiology reports will significantly reduce the burden for doctors and assist them in writing manual reports. Because of the sensitivity of the findings of the Chest X-rays (CXR) provided by the existing techniques are not adequately accurate. Therefore, producing comprehensive explanations for medical photographs remains a difficult task. A novel approach to address this issue is proposed. It is based on continuous integration of Convolution Neural Networks (CNNs) for detecting diseases and Long Short-Term Memory (LSTM) followed by Attention Mechanism for sequence generation based on those diseases. Experimental results obtained by utilizing Indiana University (IU) CXR dataset showed that the suggested model attains the current state-of-the-art efficiency as opposed to other solutions to the baseline. Some experiments were also performed without using attention block. As evaluation metric, BELU-0, BELU-1, BELU-2 and BELU-3 have been applied. The score we got without the attention Mechanism are 0.522, 0.262, 0.201, 0.119. With attention Mechanism the state of the art BELU scores are 0.580, 0.342, 0.263 and 0.155. Red heat maps also attained that not only captured the diseases present in an image, but it also expresses how these diseases are related to each other as well as their attributes and the activities they are involved in.