t5-base

Training Data

The model is pre-trained on the Colossal Clean Crawled Corpus (C4) , which was developed and released in the context of the same research paper as T5. The model was pre-trained on a on a multi-task mixture of unsupervised and supervised tasks.
Thereby, the following datasets were being used for:

Datasets used for Unsupervised denoising objective:

• C4
• Wiki-DPR

Datasets used for Supervised text-to-text language modeling objective

• Sentence acceptability judgment
  • CoLA Warstadt et al., 2018
• Sentiment analysis
  • SST-2 Socher et al., 2013
• Paraphrasing/sentence similarity
  • MRPC Dolan and Brockett, 2005
  • STS-B Ceret al., 2017
  • QQP Iyer et al., 2017
• Natural language inference
  • MNLI Williams et al., 2017
  • QNLI Rajpurkar et al.,2016
  • RTE Dagan et al., 2005
  • CB De Marneff et al., 2019
• Sentence completion
  • COPA Roemmele et al., 2011
• Word sense disambiguation
  • WIC Pilehvar and Camacho-Collados, 2018
• Question answering
  • MultiRC Khashabi et al., 2018
  • ReCoRD Zhang et al., 2018
  • BoolQ Clark et al., 2019

Training Procedure

In their abstract , the model developers write:

In this paper, we explore the landscape of transfer learning techniques for NLP by introducing a unified framework that converts every language problem into a text-to-text format. Our systematic study compares pre-training objectives, architectures, unlabeled datasets, transfer approaches, and other factors on dozens of language understanding tasks.

The framework introduced, the T5 framework, involves a training procedure that brings together the approaches studied in the paper. See the research paper for further details.

For full results for T5-Base, see the research paper , Table 14.

Testing Data, Factors & Metrics

The developers evaluated the model on 24 tasks, see the research paper for full details.