Distance Regression and Classification with Attention Network for Nuclei Instance Segmentation

Abstract

Nuclei segmentation and classification plays a major role in routine pathology workflow. Analyzing image data for finding morphological features of cells and nuclei help pathologists in diagnosis and treatment process. Automating these tasks can minimize human intervention and reduce problems related to high variability in visual features of nuclei causing different inter observer clinical outcome. Computer aided diagnosis is not straight forward because nuclei in tumor tissues often appear in clusters and are overlapping, other problems like different clinical procedures for acquiring microscopic samples, conversion to dig ital images resulting in inconsistent data and mislabeling due to manual labor work of annotating scanned whole slide images. Hence a generalized robust algorithm is hard to produce which recognizes nuclei and their types on unseen microscopic tissue biopsy images. To address these problems, I use deep learning (DL) approach and present a novel Convolutional Neural Network which is an improvement over recent state of the art CNN architecture that harnesses horizontal and vertical distance information hidden among the nuclei instances to successfully delineate challenging nuclei in digitized histology images. My pro posed methodology uses Channel and Spatial feature maps to generate relevant feature activations. These are then sequentially element wise multiplied with mainstream encoded representation maps, producing more precise and refined feature maps. As part of this work I introduce another method for improving performance on HoVer-net which uses gating technique for every bypass skip connection thereby holding down irrelevant representation of low semantic value maps from shallower layers; this technique uses attention units to suppress irrelevant noisy data that don’t contribute in learning good representations and only pass on desired salient features across the skip connection path. My contribution shows considerable improvement in classification of nuclei types and pixel-level segmenting nuclei in two major digital histology datasets i.e. CoNSeP and PanNuke fold-1. I successfully train my model on these relatively new and huge nuclei image segmentation and classification datasets and produce best results for nuclei segmentation and classification nuclei into five classes.