RSA-Net: Recursive Skip Attention Network with mask refinement for nuclei instance Segmentation and classification in histology images

Abstract

Automatic nuclei instance segmentation and classification within Hematoxylin & Eosin stained images can be considered a preliminary step in developing advanced medical systems which can provide diagnosis and prognosis to deadly diseases like cancer. How ever, the heterogeneous and crowding (overlapping and touching nuclei) nature of the cells (with large inter and intra class variability), combined with the fact that there are currently no reproducible measures to evaluate a patient’s biopsy, makes the digital pro filing of tumor micro environments even more challenging. To address these challenges, we have proposed a novel deep multi-branch CNN for simultaneous segmentation and classification of nuclei in H&E stained histopathology images. The network is composed of a shared encoder and multi-branch decoder architecture with embedded Recursive Skip Attention (RSA) blocks and a novel mask refinement step. Alongside the segmen tation and classification mask, the network also learns the horizontal and vertical pixel distances for each of the nuclei instances from their center of masses to isolate clustered and overlapping nuclei instances. The RSA blocks hooked in from the residual blocks to the decoder blocks makes the network focus more on the significant features, and that too at a very low parameter cost. The model exhibits competitive performance against SOTA methods that too on various different publicly available histology H&E stained image datasets. Additionally, we have also performed an additional mask refinement step along with the post-processing to make the network predictions even more certain.