With T5, we propose reframing all NLP tasks into a unified text-to-text-format where the input and output are always text strings, in contrast to BERT-style models that can only output either a class label or a span of the input. Our text-to-text framework allows us to use the same model, loss function, and hyperparameters on any NLP task.

Training Details

Training Data

Datasets used for Unsupervised denoising objective:

• C4
• Wiki-DPR

Datasets used for Supervised text-to-text language modeling objective

• Sentence acceptability judgment
  • CoLA Warstadt et al., 2018
• Sentiment analysis
  • SST-2 Socher et al., 2013
• Paraphrasing/sentence similarity
  • MRPC Dolan and Brockett, 2005
  • STS-B Ceret al., 2017
  • QQP Iyer et al., 2017
• Natural language inference
  • MNLI Williams et al., 2017
  • QNLI Rajpurkar et al.,2016
  • RTE Dagan et al., 2005
  • CB De Marneff et al., 2019
• Sentence completion
  • COPA Roemmele et al., 2011
• Word sense disambiguation
  • WIC Pilehvar and Camacho-Collados, 2018
• Question answering
  • MultiRC Khashabi et al., 2018
  • ReCoRD Zhang et al., 2018
  • BoolQ Clark et al., 2019

Training Procedure

In this paper, we explore the landscape of transfer learning techniques for NLP by introducing a unified framework that converts every language problem into a text-to-text format. Our systematic study compares pre-training objectives, architectures, unlabeled datasets, transfer approaches, and other factors on dozens of language understanding tasks.

Evaluation Results

Testing Data, Factors & Metrics

Model Evaluation samples

Inference samples

Sample inputs and outputs

Sample input

{
    "input_data": [
        "translate English to French: Life is so beautiful, once you learn how to live with it",
        "translate English to German: Berlin is the capital of Germany"
    ]
}

Sample output

[
  "La vie est si belle, une fois que vous apprenez à la vivre",
  "Berlin ist die Hauptstadt Deutschlands"
]