Transformers for Natural Language Processing - Second Edition

Spis treści książki

• Preface
  • Who this book is for
  • What this book covers
  • To get the most out of this book
  • Get in touch
• What are Transformers?
  • The ecosystem of transformers
    • Industry 4.0
    • Foundation models
      • Is programming becoming a sub-domain of NLP?
      • The future of artificial intelligence specialists
  • Optimizing NLP models with transformers
    • The background of transformers
  • What resources should we use?
    • The rise of Transformer 4.0 seamless APIs
    • Choosing ready-to-use API-driven libraries
    • Choosing a Transformer Model
    • The role of Industry 4.0 artificial intelligence specialists
  • Summary
  • Questions
  • References
• Getting Started with the Architecture of the Transformer Model
  • The rise of the Transformer: Attention is All You Need
    • The encoder stack
      • Input embedding
      • Positional encoding
      • Sublayer 1: Multi-head attention
      • Sublayer 2: Feedforward network
    • The decoder stack
      • Output embedding and position encoding
      • The attention layers
      • The FFN sublayer, the post-LN, and the linear layer
  • Training and performance
  • Tranformer models in Hugging Face
  • Summary
  • Questions
  • References
• Fine-Tuning BERT Models
  • The architecture of BERT
    • The encoder stack
      • Preparing the pretraining input environment
      • Pretraining and fine-tuning a BERT model
  • Fine-tuning BERT
    • Hardware constraints
    • Installing the Hugging Face PyTorch interface for BERT
    • Importing the modules
    • Specifying CUDA as the device for torch
    • Loading the dataset
    • Creating sentences, label lists, and adding BERT tokens
    • Activating the BERT tokenizer
    • Processing the data
    • Creating attention masks
    • Splitting the data into training and validation sets
    • Converting all the data into torch tensors
    • Selecting a batch size and creating an iterator
    • BERT model configuration
    • Loading the Hugging Face BERT uncased base model
    • Optimizer grouped parameters
    • The hyperparameters for the training loop
    • The training loop
    • Training evaluation
    • Predicting and evaluating using the holdout dataset
    • Evaluating using the Matthews Correlation Coefficient
    • The scores of individual batches
    • Matthews evaluation for the whole dataset
  • Summary
  • Questions
  • References
• Pretraining a RoBERTa Model from Scratch
  • Training a tokenizer and pretraining a transformer
  • Building KantaiBERT from scratch
    • Step 1: Loading the dataset
    • Step 2: Installing Hugging Face transformers
    • Step 3: Training a tokenizer
    • Step 4: Saving the files to disk
    • Step 5: Loading the trained tokenizer files
    • Step 6: Checking resource constraints: GPU and CUDA
    • Step 7: Defining the configuration of the model
    • Step 8: Reloading the tokenizer in transformers
    • Step 9: Initializing a model from scratch
      • Exploring the parameters
    • Step 10: Building the dataset
    • Step 11: Defining a data collator
    • Step 12: Initializing the trainer
    • Step 13: Pretraining the model
    • Step 14: Saving the final model (+tokenizer + config) to disk
    • Step 15: Language modeling with FillMaskPipeline
  • Next steps
  • Summary
  • Questions
  • References
• Downstream NLP Tasks with Transformers
  • Transduction and the inductive inheritance of transformers
    • The human intelligence stack
    • The machine intelligence stack
  • Transformer performances versus Human Baselines
    • Evaluating models with metrics
      • Accuracy score
      • F1-score
      • Matthews Correlation Coefficient (MCC)
    • Benchmark tasks and datasets
      • From GLUE to SuperGLUE
      • Introducing higher Human Baselines standards
      • The SuperGLUE evaluation process
    • Defining the SuperGLUE benchmark tasks
      • BoolQ
      • Commitment Bank (CB)
      • Multi-Sentence Reading Comprehension (MultiRC)
      • Reading Comprehension with Commonsense Reasoning Dataset (ReCoRD)
      • Recognizing Textual Entailment (RTE)
      • Words in Context (WiC)
      • The Winograd schema challenge (WSC)
  • Running downstream tasks
    • The Corpus of Linguistic Acceptability (CoLA)
    • Stanford Sentiment TreeBank (SST-2)
    • Microsoft Research Paraphrase Corpus (MRPC)
    • Winograd schemas
  • Summary
  • Questions
  • References
• Machine Translation with the Transformer
  • Defining machine translation
    • Human transductions and translations
    • Machine transductions and translations
  • Preprocessing a WMT dataset
    • Preprocessing the raw data
    • Finalizing the preprocessing of the datasets
  • Evaluating machine translation with BLEU
    • Geometric evaluations
    • Applying a smoothing technique
      • Chencherry smoothing
  • Translation with Google Translate
  • Translations with Trax
    • Installing Trax
    • Creating the original Transformer model
    • Initializing the model using pretrained weights
    • Tokenizing a sentence
    • Decoding from the Transformer
    • De-tokenizing and displaying the translation
  • Summary
  • Questions
  • References
• The Rise of Suprahuman Transformers with GPT-3 Engines
  • Suprahuman NLP with GPT-3 transformer models
  • The architecture of OpenAI GPT transformer models
    • The rise of billion-parameter transformer models
    • The increasing size of transformer models
      • Context size and maximum path length
    • From fine-tuning to zero-shot models
    • Stacking decoder layers
    • GPT-3 engines
  • Generic text completion with GPT-2
    • Step 9: Interacting with GPT-2
  • Training a custom GPT-2 language model
    • Step 12: Interactive context and completion examples
  • Running OpenAI GPT-3 tasks
    • Running NLP tasks online
    • Getting started with GPT-3 engines
      • Running our first NLP task with GPT-3
      • NLP tasks and examples
  • Comparing the output of GPT-2 and GPT-3
  • Fine-tuning GPT-3
    • Preparing the data
      • Step 1: Installing OpenAI
      • Step 2: Entering the API key
      • Step 3: Activating OpenAIs data preparation module
    • Fine-tuning GPT-3
      • Step 4: Creating an OS environment
      • Step 5: Fine-tuning OpenAIs Ada engine
      • Step 6: Interacting with the fine-tuned model
  • The role of an Industry 4.0 AI specialist
    • Initial conclusions
  • Summary
  • Questions
  • References
• Applying Transformers to Legal and Financial Documents for AI Text Summarization
  • Designing a universal text-to-text model
    • The rise of text-to-text transformer models
    • A prefix instead of task-specific formats
    • The T5 model
  • Text summarization with T5
    • Hugging Face
      • Hugging Face transformer resources
    • Initializing the T5-large transformer model
      • Getting started with T5
      • Exploring the architecture of the T5 model
    • Summarizing documents with T5-large
      • Creating a summarization function
      • A general topic sample
      • The Bill of Rights sample
      • A corporate law sample
  • Summarization with GPT-3
  • Summary
  • Questions
  • References
• Matching Tokenizers and Datasets
  • Matching datasets and tokenizers
    • Best practices
      • Step 1: Preprocessing
      • Step 2: Quality control
      • Continuous human quality control
    • Word2Vec tokenization
      • Case 0: Words in the dataset and the dictionary
      • Case 1: Words not in the dataset or the dictionary
      • Case 2: Noisy relationships
      • Case 3: Words in the text but not in the dictionary
      • Case 4: Rare words
      • Case 5: Replacing rare words
      • Case 6: Entailment
  • Standard NLP tasks with specific vocabulary
    • Generating unconditional samples with GPT-2
    • Generating trained conditional samples
    • Controlling tokenized data
  • Exploring the scope of GPT-3
  • Summary
  • Questions
  • References
• Semantic Role Labeling with BERT-Based Transformers
  • Getting started with SRL
    • Defining semantic role labeling
      • Visualizing SRL
    • Running a pretrained BERT-based model
      • The architecture of the BERT-based model
      • Setting up the BERT SRL environment
  • SRL experiments with the BERT-based model
  • Basic samples
    • Sample 1
    • Sample 2
    • Sample 3
  • Difficult samples
    • Sample 4
    • Sample 5
    • Sample 6
  • Questioning the scope of SRL
    • The limit of predicate analysis
    • Redefining SRL
  • Summary
  • Questions
  • References
• Let Your Data Do the Talking: Story, Questions, and Answers
  • Methodology
    • Transformers and methods
  • Method 0: Trial and error
  • Method 1: NER first
    • Using NER to find questions
      • Location entity questions
      • Person entity questions
  • Method 2: SRL first
    • Question-answering with ELECTRA
    • Project management constraints
    • Using SRL to find questions
  • Next steps
    • Exploring Haystack with a RoBERTa model
    • Exploring Q&A with a GTP-3 engine
  • Summary
  • Questions
  • References
• Detecting Customer Emotions to Make Predictions
  • Getting started: Sentiment analysis transformers
  • The Stanford Sentiment Treebank (SST)
    • Sentiment analysis with RoBERTa-large
  • Predicting customer behavior with sentiment analysis
    • Sentiment analysis with DistilBERT
    • Sentiment analysis with Hugging Faces models list
      • DistilBERT for SST
      • MiniLM-L12-H384-uncased
      • RoBERTa-large-mnli
      • BERT-base multilingual model
  • Sentiment analysis with GPT-3
  • Some Pragmatic I4.0 thinking before we leave
    • Investigating with SRL
    • Investigating with Hugging Face
    • Investigating with the GPT-3 playground
      • GPT-3 code
  • Summary
  • Questions
  • References
• Analyzing Fake News with Transformers
  • Emotional reactions to fake news
    • Cognitive dissonance triggers emotional reactions
      • Analyzing a conflictual Tweet
      • Behavioral representation of fake news
  • A rational approach to fake news
    • Defining a fake news resolution roadmap
    • The gun control debate
      • Sentiment analysis
      • Named entity recognition (NER)
      • Semantic Role Labeling (SRL)
      • Gun control SRL
      • Reference sites
    • COVID-19 and former President Trumps Tweets
      • Semantic Role Labeling (SRL)
  • Before we go
  • Summary
  • Questions
  • References
• Interpreting Black Box Transformer Models
  • Transformer visualization with BertViz
    • Running BertViz
      • Step 1: Installing BertViz and importing the modules
      • Step 2: Load the models and retrieve attention
      • Step 3: Head view
      • Step 4: Processing and displaying attention heads
      • Step 5: Model view
  • LIT
    • PCA
    • Running LIT
  • Transformer visualization via dictionary learning
    • Transformer factors
    • Introducing LIME
    • The visualization interface
  • Exploring models we cannot access
  • Summary
  • Questions
  • References
• From NLP to Task-Agnostic Transformer Models
  • Choosing a model and an ecosystem
  • The Reformer
    • Running an example
  • DeBERTa
    • Running an example
  • From Task-Agnostic Models to Vision Transformers
    • ViT Vision Transformers
      • The Basic Architecture of ViT
      • Vision transformers in code
    • CLIP
      • The Basic Architecture of CLIP
      • CLIP in code
    • DALL-E
      • The Basic Architecture of DALL-E
      • DALL-E in code
  • An expanding universe of models
  • Summary
  • Questions
  • References
• The Emergence of Transformer-Driven Copilots
  • Prompt engineering
    • Casual English with a meaningful context
    • Casual English with a metonymy
    • Casual English with an ellipsis
    • Casual English with vague context
    • Casual English with sensors
    • Casual English with sensors but no visible context
    • Formal English conversation with no context
    • Prompt engineering training
  • Copilots
    • GitHub Copilot
    • Codex
  • Domain-specific GPT-3 engines
    • Embedding2ML
      • Step 1: Installing and importing OpenAI
      • Step 2: Loading the dataset
      • Step 3: Combining the columns
      • Step 4: Running the GPT-3 embedding
      • Step 5: Clustering (k-means clustering) with the embeddings
      • Step 6: Visualizing the clusters (t-SNE)
    • Instruct series
    • Content filter
  • Transformer-based recommender systems
    • General-purpose sequences
    • Dataset pipeline simulation with RL using an MDP
      • Training customer behaviors with an MDP
      • Simulating consumer behavior with an MDP
      • Making recommendations
  • Computer vision
  • Humans and AI copilots in metaverses
    • From looking at to being in
  • Summary
  • Questions
  • References
• Appendix I Terminology of Transformer Models
  • Stack
  • Sublayer
  • Attention heads
• Appendix II Hardware Constraints for Transformer Models
  • The Architecture and Scale of Transformers
  • Why GPUs are so special
  • GPUs are designed for parallel computing
  • GPUs are also designed for matrix multiplication
  • Implementing GPUs in code
  • Testing GPUs with Google Colab
  • Google Colab Free with a CPU
    • Google Colab Free with a GPU
  • Google Colab Pro with a GPU
• Appendix III Generic Text Completion with GPT-2
  • Step 1: Activating the GPU
  • Step 2: Cloning the OpenAI GPT-2 repository
  • Step 3: Installing the requirements
  • Step 4: Checking the version of TensorFlow
  • Step 5: Downloading the 345M-parameter GPT-2 model
  • Steps 6-7: Intermediate instructions
  • Steps 7b-8: Importing and defining the model
  • Step 9: Interacting with GPT-2
  • References
• Appendix IV Custom Text Completion with GPT-2
  • Training a GPT-2 language model
    • Step 1: Prerequisites
    • Steps 2 to 6: Initial steps of the training process
    • Step 7: The N Shepperd training files
    • Step 8: Encoding the dataset
    • Step 9: Training a GPT-2 model
    • Step 10: Creating a training model directory
    • Step 11: Generating unconditional samples
    • Step 12: Interactive context and completion examples
  • References
• Appendix V Answers to the Questions
  • Chapter 1, What are Transformers?
  • Chapter 2, Getting Started with the Architecture of the Transformer Model
  • Chapter 3, Fine-Tuning BERT Models
  • Chapter 4, Pretraining a RoBERTa Model from Scratch
  • Chapter 5, Downstream NLP Tasks with Transformers
  • Chapter 6, Machine Translation with the Transformer
  • Chapter 7, The Rise of Suprahuman Transformers with GPT-3 Engines
  • Chapter 8, Applying Transformers to Legal and Financial Documents for AI Text Summarization
  • Chapter 9, Matching Tokenizers and Datasets
  • Chapter 10, Semantic Role Labeling with BERT-Based Transformers
  • Chapter 11, Let Your Data Do the Talking: Story, Questions, and Answers
  • Chapter 12, Detecting Customer Emotions to Make Predictions
  • Chapter 13, Analyzing Fake News with Transformers
  • Chapter 14, Interpreting Black Box Transformer Models
  • Chapter 15, From NLP to Task-Agnostic Transformer Models
  • Chapter 16, The Emergence of Transformer-Driven Copilots
• Other Books You May Enjoy
• Index

pokaż cały spis treści

ukryj recenzje