distilbert-base-uncased-distilled-squad
DistilBERT model was proposed in the blog post Smaller, faster, cheaper, lighter: Introducing DistilBERT, adistilled version of BERT , and the paper DistilBERT, adistilled version of BERT: smaller, faster, cheaper and lighter . DistilBERT is a small, fast, cheap and light Transformer model trained by distilling BERT base. It has 40% less parameters than bert-base-uncased, runs 60% faster while preserving over 95% of BERT's performances as measured on the GLUE language understanding benchmark.
This model is a fine-tune checkpoint of DistilBERT-base-uncased , fine-tuned using (a second step of) knowledge distillation on SQuAD v1.1 .
Training Data
The distilbert-base-uncased model model describes it's training data as:
DistilBERT pretrained on the same data as BERT, which is BookCorpus , a dataset consisting of 11,038 unpublished books and English Wikipedia (excluding lists, tables and headers).
To learn more about the SQuAD v1.1 dataset, see the SQuAD v1.1 data card .
Training Procedure
Preprocessing
See the distilbert-base-uncased model card for further details.
Pretraining
See the distilbert-base-uncased model card for further details.
As discussed in the model repository
This model reaches a F1 score of 86.9 on the [SQuAD v1.1] dev set (for comparison, Bert bert-base-uncased version reaches a F1 score of 88.5).
