Context-aware layout to image generation with enhanced object appearance

Sen He, Wentong Liao, Michael Ying Yang, Yongxin Yang, Yi-zhe Song, Bodo Rosenhahn, Tao Xiang

Research output: Chapter in Book/Report/Conference proceedingConference contributionAcademicpeer-review

Abstract

A layout to image (L2I) generation model aims to generate a complicated image containing multiple objects (things) against natural background (stuff), conditioned on a given layout. Built upon the recent advances in generative adversarial networks (GANs), existing L2I models have made great progress. However, a close inspection of their generated images reveals two major limitations: (1) the object-to-object as well as object-to-stuff relations are often broken and (2) each object’s appearance is typically distorted lacking the key defining characteristics associated with the object class. We argue that these are caused by the lack of context-aware object and stuff feature encoding in their generators, and location-sensitive appearance representation in their discriminators. To address these limitations, two new modules are proposed in this work. First, a context-aware feature transformation module is introduced in the generator to ensure that the generated feature encoding of either object or stuff is aware of other coexisting objects/stuff in the scene. Second, instead of feeding location-insensitive image features to the discriminator, we use the Gram matrix computed from the feature maps of the generated object images to preserve location-sensitive information, resulting in much enhanced object appearance. Extensive experiments show that the proposed method achieves state-of-the-art performance on the COCO-Thing-Stuff and Visual Genome benchmarks. Code available at: https://github.com/wtliao/layout2img.
Original languageEnglish
Title of host publication2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
PublisherIEEE
Pages15044-15053
Number of pages10
ISBN (Electronic)978-1-6654-4509-2
ISBN (Print)978-1-6654-4510-8
DOIs
Publication statusPublished - 13 Nov 2021
Event2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) - Nashville, TN, USA
Duration: 20 Jun 202125 Jun 2021

Conference

Conference2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
Period20/06/2125/06/21

Fingerprint

Dive into the research topics of 'Context-aware layout to image generation with enhanced object appearance'. Together they form a unique fingerprint.

Cite this