Skip to content

sudharsan13296/Getting-Started-with-Google-BERT

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

23 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Build and train state-of-the-art natural language processing models using BERT

About the book

Book Cover

BERT (bidirectional encoder representations from transformer) has revolutionized the world of natural language processing (NLP) with promising results. This book is an introductory guide that will help you get to grips with Google's BERT architecture. With a detailed explanation of the transformer architecture, this book will help you understand how the transformer's encoder and decoder work.

You'll explore the BERT architecture by learning how the BERT model is pre-trained and how to use pre-trained BERT for downstream tasks by fine-tuning it for NLP tasks such as sentiment analysis and text summarization with the Hugging Face transformers library. As you advance, you'll learn about different variants of BERT such as ALBERT, RoBERTa, and ELECTRA, and look at SpanBERT, which is used for NLP tasks like question answering. You'll also cover simpler and faster BERT variants based on knowledge distillation such as DistilBERT and TinyBERT.

The book takes you through MBERT, XLM, and XLM-R in detail and then introduces you to sentence-BERT, which is used for obtaining sentence representation. Finally, you'll discover domain-specific BERT models such as BioBERT and ClinicalBERT, and discover an interesting variant called VideoBERT.

Get the book


Clone the repo and run in Google Colab

  • 2.1. Basic idea of BERT
  • 2.2. Working of BERT
  • 2.3. Configuration of BERT
  • 2.4. Pre-training the BERT
  • 2.5. Pre-training strategies
  • 2.6. Pre-training procedure
  • 2.7. Subword tokenization algorithms
  • 2.8. Byte pair encoding
  • 2.9. Byte-level byte pair encoding
  • 2.10. WordPiece
  • 5.1. Knowledge distillation
  • 5.2. DistilBERT - distilled version of BERT
  • 5.3. Training the DistilBERT
  • 5.4. TinyBERT
  • 5.5. Teacher-student architecture
  • 5.6. Training the TinyBERT
  • 5.7. Transferring knowledge from BERT to neural network
  • 5.8. Teacher-student architecture
  • 5.9. Training the student network
  • 5.10. Data augmentation method
  • 6.1. Text summarization
  • 6.2. Fine-tuning BERT for text summarization
  • 6.3. Extractive summarization using BERT
  • 6.4. Abstractive summarization using BERT
  • 6.5. Understanding ROUGE evaluation metric
  • 6.6. Performance of BERTSUM model
  • 6.7. Training the BERTSUM model