← ML Research Wiki / 2302.08453

T2I-Adapter: Learning Adapters to Dig Out More Controllable Ability for Text-to-Image Diffusion Models

Chong Mou [email protected] Shenzhen Graduate School Peking University ARC Lab Tencent PCG, Xintao Wang ARC Lab Tencent PCG, Liangbin Xie [email protected] ARC Lab Tencent PCG University of Macau Shenzhen Institute of Advanced Technology, Yanze Wu [email protected] ARC Lab Tencent PCG, Jian Zhang [email protected] Shenzhen Graduate School Peking University, Zhongang Qi [email protected] ARC Lab Tencent PCG, Ying Shan [email protected] ARC Lab Tencent PCG (2023)

Paper Information
arXiv ID
2302.08453
Venue
AAAI Conference on Artificial Intelligence
Domain
computer vision, natural language processing
SOTA Claim
Yes
Code
Available
Reproducibility
8/10
Contents

Abstract

The incredible generative ability of large-scale text-to-image (T2I) models has demonstrated strong power of learning complex structures and meaningful semantics.However, relying solely on text prompts cannot fully take advantage of the knowledge learned by the model, especially when flexible and accurate controlling (e.g., structure and color) is needed.In this paper, we aim to "dig out" the capabilities that T2I models have implicitly learned, and then explicitly use them to control the generation more granularly.Specifically, we propose to learn low-cost T2I-Adapters to align internal knowledge in T2I models with external control signals, while freezing the original large T2I models.In this way, we can train various adapters according to different conditions, achieving rich control and editing effects in the color and structure of the generation results.Further, the proposed T2I-Adapters have attractive properties of practical value, such as composability and generalization ability.Extensive experiments demonstrate that our T2I-Adapter has promising generation quality and a wide range of applications.Our code is available at https://github.com/TencentARC/T2I-Adapter.

Summary

The paper presents T2I-Adapter, a low-cost method aimed at improving controllability in text-to-image (T2I) diffusion models by aligning the internal knowledge of these models with external control signals while keeping the original model parameters unchanged. The authors propose learning adapters that allow for various controls over the image generation process, including spatial structure and color. T2I-Adapters are designed to be simple, flexible, and composable, allowing for an enhanced generation quality in T2I models. Existing models like Stable Diffusion (SD) are utilized for demonstrating the effectiveness of T2I-Adapters. The paper also compares their approach against notable methods like ControlNet to validate performance. The findings assert that T2I-Adapters provide a practical solution for effective and granular control in image generation tasks.

Methods

This paper employs the following methods:

  • T2I-Adapter
  • Stable Diffusion

Models Used

  • Stable Diffusion

Datasets

The following datasets were used in this research:

  • COCO

Evaluation Metrics

  • FID
  • CLIP score

Results

  • Promising generation quality
  • Capable of providing accurate controllable guidance
  • Effective across various conditions with composability features

Limitations

The authors identified the following limitations:

  • Manual adjustment required for multi-adapter control combinations.

Technical Requirements

  • Number of GPUs: 4
  • GPU Type: NVIDIA Tesla V100 32GB

Keywords

text-to-image diffusion models T2I-Adapter controllable generation diffusion models structure control color control

Papers Using Similar Methods

External Resources

Funding: National Natural Science Foundation of China
References: 60
Influential Citations: 106