Transformers for Natural Language Processing

Spis treści książki

• Preface
  • Who this book is for
  • What this book covers
  • To get the most out of this book
  • Get in touch
• Getting Started with the Model Architecture of the Transformer
  • The background of the Transformer
  • The rise of the Transformer: Attention Is All You Need
    • The encoder stack
      • Input embedding
      • Positional encoding
      • Sub-layer 1: Multi-head attention
      • Sub-layer 2: Feedforward network
    • The decoder stack
      • Output embedding and position encoding
      • The attention layers
      • The FFN sub-layer, the Post-LN, and the linear layer
  • Training and performance
    • Before we end the chapter
  • Summary
  • Questions
  • References
• Fine-Tuning BERT Models
  • The architecture of BERT
    • The encoder stack
      • Preparing the pretraining input environment
    • Pretraining and fine-tuning a BERT model
  • Fine-tuning BERT
    • Activating the GPU
    • Installing the Hugging Face PyTorch interface for BERT
    • Importing the modules
    • Specifying CUDA as the device for torch
    • Loading the dataset
    • Creating sentences, label lists, and adding BERT tokens
    • Activating the BERT tokenizer
    • Processing the data
    • Creating attention masks
    • Splitting data into training and validation sets
    • Converting all the data into torch tensors
    • Selecting a batch size and creating an iterator
    • BERT model configuration
    • Loading the Hugging Face BERT uncased base model
    • Optimizer grouped parameters
    • The hyperparameters for the training loop
    • The training loop
    • Training evaluation
    • Predicting and evaluating using the holdout dataset
    • Evaluating using Matthews Correlation Coefficient
    • The score of individual batches
    • Matthews evaluation for the whole dataset
  • Summary
  • Questions
  • References
• Pretraining a RoBERTa Model from Scratch
  • Training a tokenizer and pretraining a transformer
  • Building KantaiBERT from scratch
    • Step 1: Loading the dataset
    • Step 2: Installing Hugging Face transformers
    • Step 3: Training a tokenizer
    • Step 4: Saving the files to disk
    • Step 5: Loading the trained tokenizer files
    • Step 6: Checking resource constraints: GPU and CUDA
    • Step 7: Defining the configuration of the model
    • Step 8: Reloading the tokenizer in transformers
    • Step 9: Initializing a model from scratch
      • Exploring the parameters
    • Step 10: Building the dataset
    • Step 11: Defining a data collator
    • Step 12: Initializing the trainer
    • Step 13: Pretraining the model
    • Step 14: Saving the final model (+tokenizer + config) to disk
    • Step 15: Language modeling with FillMaskPipeline
  • Next steps
  • Summary
  • Questions
  • References
• Downstream NLP Tasks with Transformers
  • Transduction and the inductive inheritance of transformers
    • The human intelligence stack
    • The machine intelligence stack
  • Transformer performances versus Human Baselines
    • Evaluating models with metrics
      • Accuracy score
      • F1-score
      • Matthews Correlation Coefficient (MCC)
    • Benchmark tasks and datasets
      • From GLUE to SuperGLUE
      • Introducing higher Human Baseline standards
      • The SuperGLUE evaluation process
    • Defining the SuperGLUE benchmark tasks
      • BoolQ
      • Commitment Bank (CB)
      • Multi-Sentence Reading Comprehension (MultiRC)
      • Reading Comprehension with Commonsense Reasoning Dataset (ReCoRD)
      • Recognizing Textual Entailment (RTE)
      • Words in Context (WiC)
      • The Winograd Schema Challenge (WSC)
  • Running downstream tasks
    • The Corpus of Linguistic Acceptability (CoLA)
    • Stanford Sentiment TreeBank (SST-2)
    • Microsoft Research Paraphrase Corpus (MRPC)
    • Winograd schemas
  • Summary
  • Questions
  • References
• Machine Translation with the Transformer
  • Defining machine translation
    • Human transductions and translations
    • Machine transductions and translations
  • Preprocessing a WMT dataset
    • Preprocessing the raw data
    • Finalizing the preprocessing of the datasets
  • Evaluating machine translation with BLEU
    • Geometric evaluations
    • Applying a smoothing technique
      • Chencherry smoothing
  • Translations with Trax
    • Installing Trax
    • Creating a Transformer model
    • Initializing the model using pretrained weights
    • Tokenizing a sentence
    • Decoding from the Transformer
    • De-tokenizing and displaying the translation
  • Summary
  • Questions
  • References
• Text Generation with OpenAI GPT-2 and GPT-3 Models
  • The rise of billion-parameter transformer models
    • The increasing size of transformer models
      • Context size and maximum path length
  • Transformers, reformers, PET, or GPT?
    • The limits of the original Transformer architecture
      • Running BertViz
    • The Reformer
    • Pattern-Exploiting Training (PET)
      • The philosophy of Pattern-Exploiting Training (PET)
  • Its time to make a decision
  • The architecture of OpenAI GPT models
    • From fine-tuning to zero-shot models
    • Stacking decoder layers
  • Text completion with GPT-2
    • Step 1: Activating the GPU
    • Step 2: Cloning the OpenAI GPT-2 repository
    • Step 3: Installing the requirements
    • Step 4: Checking the version of TensorFlow
    • Step 5: Downloading the 345M parameter GPT-2 model
    • Steps 6-7: Intermediate instructions
    • Steps 7b-8: Importing and defining the model
    • Step 9: Interacting with GPT-2
  • Training a GPT-2 language model
    • Step 1: Prerequisites
    • Steps 2 to 6: Initial steps of the training process
    • Step 7: The N Shepperd training files
    • Step 8: Encoding the dataset
    • Step 9: Training the model
    • Step 10: Creating a training model directory
  • Context and completion examples
  • Generating music with transformers
  • Summary
  • Questions
  • References
• Applying Transformers to Legal and Financial Documents for AI Text Summarization
  • Designing a universal text-to-text model
    • The rise of text-to-text transformer models
    • A prefix instead of task-specific formats
    • The T5 model
  • Text summarization with T5
    • Hugging Face
      • Hugging Face transformer resources
    • Initializing the T5-large transformer model
      • Getting started with T5
      • Exploring the architecture of the T5 model
    • Summarizing documents with T5-large
      • Creating a summarization function
      • A general topic sample
      • The Bill of Rights sample
      • A corporate law sample
  • Summary
  • Questions
  • References
• Matching Tokenizers and Datasets
  • Matching datasets and tokenizers
    • Best practices
      • Step 1: Preprocessing
      • Step 2: Post-processing
      • Continuous human quality control
    • Word2Vec tokenization
      • Case 0: Words in the dataset and the dictionary
      • Case 1: Words not in the dataset or the dictionary
      • Case 2: Noisy relationships
      • Case 3: Rare words
      • Case 4: Replacing rare words
      • Case 5: Entailment
  • Standard NLP tasks with specific vocabulary
    • Generating unconditional samples with GPT-2
      • Controlling tokenized data
    • Generating trained conditional samples
  • T5 Bill of Rights Sample
    • Summarizing the Bill of Rights, version 1
    • Summarizing the Bill of Rights, version 2
  • Summary
  • Questions
  • References
• Semantic Role Labeling with BERT-Based Transformers
  • Getting started with SRL
    • Defining Semantic Role Labeling
      • Visualizing SRL
    • Running a pretrained BERT-based model
      • The architecture of the BERT-based model
      • Setting up the BERT SRL environment
  • SRL experiments with the BERT-based model
  • Basic samples
    • Sample 1
    • Sample 2
    • Sample 3
  • Difficult samples
    • Sample 4
    • Sample 5
    • Sample 6
  • Summary
  • Questions
  • References
• Let Your Data Do the Talking: Story, Questions, and Answers
  • Methodology
    • Transformers and methods
  • Method 0: Trial and error
  • Method 1: NER first
    • Using NER to find questions
      • Location entity questions
      • Person entity questions
  • Method 2: SRL first
    • Question-answering with ELECTRA
    • Project management constraints
    • Using SRL to find questions
  • Next steps
    • Exploring Haystack with a RoBERTa model
  • Summary
  • Questions
  • References
• Detecting Customer Emotions to Make Predictions
  • Getting started: Sentiment analysis transformers
  • The Stanford Sentiment Treebank (SST)
    • Sentiment analysis with RoBERTa-large
  • Predicting customer behavior with sentiment analysis
    • Sentiment analysis with DistilBERT
    • Sentiment analysis with Hugging Face's models list
      • DistilBERT for SST
      • MiniLM-L12-H384-uncased
      • RoBERTa-large-mnli
      • BERT-base multilingual model
  • Summary
  • Questions
  • References
• Analyzing Fake News with Transformers
  • Emotional reactions to fake news
    • Cognitive dissonance triggers emotional reactions
      • Analyzing a conflictual Tweet
      • Behavioral representation of fake news
  • A rational approach to fake news
    • Defining a fake news resolution roadmap
    • Gun control
      • Sentiment analysis
      • Named entity recognition (NER)
      • Semantic Role Labeling (SRL)
      • Reference sites
    • COVID-19 and former President Trump's Tweets
      • Semantic Role Labeling (SRL)
  • Before we go
    • Looking for the silver bullet
    • Looking for reliable training methods
  • Summary
  • Questions
  • References
• Appendix: Answers to the Questions
  • Chapter 1, Getting Started with the Model Architecture of the Transformer
  • Chapter 2, Fine-Tuning BERT Models
  • Chapter 3, Pretraining a RoBERTa Model from Scratch
  • Chapter 4, Downstream NLP Tasks with Transformers
  • Chapter 5, Machine Translation with the Transformer
  • Chapter 6, Text Generation with OpenAI GPT-2 and GPT-3 Models
  • Chapter 7, Applying Transformers to Legal and Financial Documents for AI Text Summarization
  • Chapter 8, Matching Tokenizers and Datasets
  • Chapter 9, Semantic Role Labeling with BERT-Based Transformers
  • Chapter 10, Let Your Data Do the Talking: Story, Questions, and Answers
  • Chapter 11, Detecting Customer Emotions to Make Predictions
  • Chapter 12, Analyzing Fake News with Transformers
• Other Books You May Enjoy
• Index

pokaż cały spis treści

ukryj recenzje