LayoutGPT

Abstract

Attaining a high degree of user controllability in visual generation often requires intricate, fine-grained inputs like layouts. However, such inputs impose a substantial burden on users when compared to simple text inputs. To address the issue, we study how Large Language Models (LLMs) can serve as visual planners by generating layouts from text conditions, and thus collaborate with visual generative models.

We propose LayoutGPT, a method to compose in-context visual demonstrations in style sheet language to enhance the visual planning skills of LLMs. LayoutGPT can generate plausible layouts in multiple domains, ranging from 2D images to 3D indoor scenes. LayoutGPT also shows superior performance in converting challenging language concepts like numerical and spatial relations to layout arrangements for faithful text-to-image generation.

When combined with a downstream image generation model, LayoutGPT outperforms text-to-image models/systems by 20-40% and achieves comparable performance as human users in designing visual layouts for numerical and spatial correctness. Lastly, LayoutGPT achieves comparable performance to supervised methods in 3D indoor scene synthesis, demonstrating its effectiveness and potential in multiple visual domains.

2D image layouts

Faithfulness in Numerical and Spatial Concepts

LayoutGPT can apply the numerical reasoning skills of LLMs into layout generation and learn spatial concepts through in-context demonstrations.

Flexible Application Scenarios

Two natural advantages of using LLMs for image layout generation:
(1) Attribute Binding: assign correct attributes to the bounding boxes
(2) Text-based inpainting: imagine and expand the underspecified description of certain objects.

3D Scene Synthesis

LayoutGPT shows comparable performance as supervised methods in indoor scene generation conditioned on room type and floor plan size.

Application: Scene Completion

The autoregressive manner of LLMs enables LayoutGPT to complete a partial scene.

BibTeX

@article{feng2023layoutgpt,
  title={LayoutGPT: Compositional Visual Planning and Generation with Large Language Models},
  author={Feng, Weixi and Zhu, Wanrong and Fu, Tsu-jui and Jampani, Varun and Akula, Arjun and He, Xuehai and Basu, Sugato and Wang, Xin Eric and Wang, William Yang},
  journal={arXiv preprint arXiv:2305.15393},
  year={2023}
}

LayoutGPT: Compositional Visual Planning and Generation with Large Language Models

LayoutGPT for image layout generation based on text inputs.

LayoutGPT for 3D indoor scene synthesis.