Natural Image Synthesis Using Scene Graphs

Abstract

Complex scene graphs are still battling to create fewer artefacts and distinct instances of an object when employed for image generation tasks. A closer look at the generated images found in the literature reveals that the appearance of each object is often distorted and devoid of the essential distinguishing characteristics connected with the object class. We believe; it is due to the loss of sensitive information during the process where each node and edge of the graph transform into their respective encodings. We aim to address this issue by employing an autoencoder-based graph convolutional network to reduce information loss. Our model encodes topological structure presenting relationship and object content of the scene graph into meaningful compact representation. Our approach significantly denoise the learned embeddings and promises to carry more relevant and discriminative information than embeddings obtained directly from a scene graph via graph convolutional networks. These newly learnt embeddings are then transmitted to the appropriate networks to generate bounding boxes and segmentation masks, which are subsequently utilised in the scene layout generating process. The cascaded refinement network uses scene layout to synthesize an image. Using COCO-Stuff, we evaluate the ability of our system in terms of the inception score and human subjective judgement.