ML and Generative AI in the Data Lakehouse. Building and Deploying AI Applications at Scale Bennie Haelen

Dostępność produktu

Produkt został zweryfikowany pod kątem ułatwień dostępu więcej przeczytasz w deklaracji dostępności.

Spis treści książki

• Preface
  • Who Should Read This Book
  • Why I Wrote This Book
  • Navigating This Book
  • Conventions Used in This Book
  • Using Code Examples
  • OReilly Online Learning
  • How to Contact Us
  • Acknowledgments
• 1. An Overview of Machine Learning, AI, and GenAI
  • What Is AI?
    • The Evolution of AI
      • Traditional AI: rule-based systems (1950s to 1980s)
      • Machine learning: learning from data (1990s to 2000s)
      • Deep learning: neural networks at scale (2010s)
      • GenAI: Creating new content (2020s)
      • Large language models: AI that understands language
  • The Subdomains of ML
    • Supervised Learning
    • Unsupervised Learning
    • Semi-Supervised Learning
    • Reinforcement Learning
  • An Overview of GenAI
    • Types of Generative Models
      • Generative adversarial networks
      • Variational autoencoders
      • Transformer-based models
    • GenAI Applications and Use Cases
  • Responsible AI
    • Data Privacy
    • Accountability
    • Transparency
    • Explainability
    • Privacy Risks
    • Societal Impact
    • Content Integrity and Accuracy
  • Summary
• 2. The Databricks Data Intelligence Platform
  • Why the Databricks Platform?
    • Unified Lakehouse Architecture
    • Collaborative Tools and Managed Infrastructure
    • Comprehensive ML Ecosystem
    • Training, Inference, and the GenAI Paradigm Shift
    • Enterprise Security and Governance
  • Databricks Platform Overview
    • Data Storage and Reliability
    • Data Governance and Security
    • Data Engineering and Pipelines
    • Analytics and Business Intelligence
    • ML and AI
  • The Databricks Runtime
    • Optimized Spark Engine
    • Integrated ML Libraries
    • Language Support
    • Advanced Analytics and AI Capabilities
    • Runtime Summary
  • Clusters, Workspaces, and Notebooks
    • Understanding Databricks Clusters
      • Interactive development workflow
      • Automated execution workflow
      • Key features of Databricks clusters
    • Navigating Databricks Workspaces
    • Leveraging Databricks Notebooks
      • Features of Databricks notebooks
      • Notebooks in ML, AI, and GenAI
    • Integrating Clusters, Workspaces, and Notebooks
  • Setting Up an ML Environment
    • Understanding the Databricks ML Workspace
    • Prerequisites
    • Creating an Account and a Workspace
      • Creating a Databricks account
      • Setting up a Databricks workspace
    • Setting Up Clusters for ML
      • Accessing your Databricks workspace
      • Simple form versus advanced configuration
      • Selecting the cluster type
      • Understanding cluster topologies: multinode versus single node
      • Selecting the cluster policy
      • Understanding cluster access modes
    • Runtime Selection
      • Long-term support (LTS) versus non-LTS Runtimes
      • Standard Runtime versus ML Runtime
      • GPU support
      • Photon acceleration
    • Driver and Worker Nodes
      • Configuring driver and worker nodes
        • Selecting instance types
        • Driver node configuration
        • Worker node configuration
        • Configuring cluster size
      • Serverless compute
      • When to choose serverless versus traditional clusters
      • Final configuration
    • Extending the ML Runtime with Additional Libraries
      • Specialized and cutting-edge libraries
      • Custom and proprietary libraries
      • Methods for adding libraries
  • Summary
• 3. An Introduction to ML on Databricks
  • The End-to-End ML Environment
    • Data Preparation
      • Delta tables
      • Feature Engineering in Unity Catalog
    • Model Training
    • Model Creation and Validation
    • Model Productionalization
  • Managing the ML Lifecycle with MLflow
    • Challenge 1: Experiment Tracking and Reproducibility
    • Challenge 2: Model Versioning and Lineage
    • Challenge 3: Collaboration Across Teams
    • Challenge 4: Model Packaging and Deployment
    • MLflow in the MLOps Landscape
    • MLflow Components
      • MLflow Tracking
      • MLflow Models
      • MLflow Model Serving
      • MLflow Projects
      • MLflow Model Registry
    • Example Workflow
  • Summary
• 4. End-to-End ML with MLflow
  • ML in Context
    • Matching ML Approaches to Business Problems
    • Focus on Implementation, Not Model Selection
    • What You Will Build
  • Exploratory Data Analysis
    • Loading the Dataset
    • Creating a Volume
    • Uploading the Dataset to a Volume
    • Exploring the Dataset
      • Initial data inspection
      • Key questions explored
        • Question: From what country do the most guests originate?
        • Question: How much do guests pay per room per night?
        • Question: How does price vary throughout the year?
        • Question: Which are the busiest months?
        • Question: How long do guests typically stay?
      • Hypothesis validation
        • Hypothesis 1: City hotels have higher cancellation rates than resort hotels
        • Hypothesis 2: Earlier bookings (longer lead time) lead to more cancellations
        • Hypothesis 3: Repeat guests are less likely to cancel
        • Hypothesis 4: Higher previous cancellations predict current cancellation
        • Hypothesis 5: Room type mismatch leads to cancellations
        • Hypothesis 6: Refundable bookings have higher cancellations
      • Correlation analysis
      • Key takeaways for feature engineering
  • Feature Engineering
    • Handling Missing Values
    • Data Cleansing
      • Encoding categorical values
      • Correlation-based feature reduction
    • Saving to Unity Catalog
  • Model Training
    • Creating an MLflow Experiment
    • Model Building
      • Reading the feature table from Unity Catalog
      • Preparing training and test data
      • Defining the feature matrix and target variable
      • Splitting into training and test sets
      • Scaling the feature matrix
      • Visualizing feature distributions after scaling
    • Model Training with MLflow Tracking
      • Setting the MLflow experiment
      • Enabling MLflow autologging
      • Training a logistic regression model
        • Additional model runs
        • Comparing results and registering the best model
  • Model Predictions
    • Preparing the Test Data
    • Loading the Model from the Unity Catalog Registry
    • Loading as a Native scikit-learn Model
    • Inspecting Feature Importances
    • Loading the Model as a Spark UDF
    • Loading the Model as a Generic PyFunc Model
    • When to Use Each Loading Method
  • Model Deployment
    • Verifying the Registered Model
    • Reviewing Model Metadata and Lineage
    • Managing the Model Lifecycle with Aliases
    • Deploying to a Model Serving Endpoint
    • Querying the Endpoint
    • Updating the Endpoint with a New Version
  • Model Inferencing
    • Preparing the Test Data
    • Retrieving Workspace Credentials
    • Defining the Scoring Function
    • Loading the Champion Model for Local Comparison
    • Calling the Endpoint and Comparing Predictions
    • Verifying Prediction Consistency
    • Evaluating Accuracy and Prediction Distribution
  • Summary
• 5. Feature Engineering in the Unity Catalog
  • The Role of Feature Engineering in ML
    • Delta Tables: The Foundation of Feature Engineering
    • Feature Tables: The Core of the Feature Store
    • Integration with Unity Catalog
  • Feature Engineering in the Lakehouse
    • Offline and Online Stores
    • Feature Consistency and Reusability
  • ML Workflow with the Feature Store
    • Step 1: Creating and Managing Feature Tables
    • Step 2: Training with Features from the Feature Store
    • Step 3: Real-Time Inference
  • Example: LendingClub Feature Store
    • Introduction to the LendingClub Dataset
    • Approach
    • Step 1: Loading and Preprocessing the LendingClub Dataset
    • Step 2: Build Your Feature Table
      • Initializing the FeatureEngineeringClient
      • Phase 1: Selecting and writing the initial features
      • Phase 2: Setting up Unity Catalog for feature storage
      • Creating the feature table
      • Merging additional features into the feature store
    • Step 3: Building the Training Dataset Using FeatureEngineeringClient
      • Defining the label DataFrame
      • Defining feature lookups
      • Creating the training set
      • Loading and preparing the training DataFrame
    • Step 4: Building the Model with Features from the Feature Table
      • Splitting the dataset
      • Choosing the Random Forest classifier
      • Training the model
      • Making predictions and evaluating performance
      • Inferring the model signature
    • Step 5: Model Visualizations
      • The feature importance plot
      • The confusion matrix
      • Target variable distribution
      • ROC and precision-recall curves
    • Step 6: Logging and Registering the Model in Unity Catalog
      • Setting the Registry URI
      • Defining the Unity Catalog model name
      • Collecting metrics for logging
      • Viewing the feature table and model in Catalog Explorer
  • Summary
• 6. ML at Scale
  • The Role of Spark in ML
    • Scaling ML with Spark
    • Distributed Data Processing
    • pandas API on Spark
    • Using Spark MLlib
    • Distributed Deep Learning: TorchDistributor, DeepSpeed, and Ray
    • Parallelized Model Training
    • Spark Structured Streaming for Real-Time ML
    • Hyperparameter Tuning with Optuna, Hyperopt, and Ray Tune
  • The Role of Delta Lake in Scalable ML
    • ACID Transactions and Schema Enforcement
    • Time Travel and Data Versioning
    • Efficient Incremental Data Processing
    • Synergy with Spark and Databricks
  • Platform Capabilities for ML at Scale
    • The Shift Away from AutoML
      • What AutoML did and what replaces it
      • Baseline model generation
      • Automated model selection across algorithms
    • The Broader Mosaic AI Direction
    • Accelerating ML Workloads with Photon
    • MLflow Integration at Scale
    • Serverless Compute for ML Workloads
    • Unity Catalog Integration for ML at Scale
  • Cost Management at Scale
  • ML at Scale Use Case
    • Step 1: Data Loading and Initial Type Conversions
      • Downloading the dataset
      • Loading the data
      • Getting an estimate of the row count
      • Performing type conversions
    • Step 2: Data Preprocessing
      • Adding a trip duration column
      • Filtering out trips with unrealistic durations or distances
      • Drop rows with missing critical values
    • Step 3: Feature Engineering
      • Extracting temporal features
      • Imputing missing values in numerical features
      • Handling missing values in categorial features
    • Step 4: Feature Transformations and Vectorization
      • Setting up indexers and encoders
      • Setting up the vector assembler
      • Executing the indexers
      • Executing the encoders
      • Assembling the features column
      • Finalizing the dataset
    • Step 5: Efficient Storage with Delta Lake
      • Writing to Delta Lake
      • Optimizing the Delta table
    • Step 6: Model Training at Scale
      • Reading back from Delta and splitting the data
      • Training a Random Forest regressor
      • Evaluating the baseline model
    • Step 7: Hyperparameter Tuning with TrainValidationSplit
    • Step 8: Distributed Hyperparameter Tuning with Optuna
      • Installing and importing Optuna
      • Defining the objective function
      • Running the Optuna study
      • Inspecting the results
      • Visualize the result
      • Extracting the best hyperparameters
    • Step 9: Log and Register the Best Model in Unity Catalog
      • Defining the Unity Catalog model name
      • Training the best model and registering in Unity Catalog
      • Assigning the Champion alias
      • Loading and verifying the registered model
    • Use Case Results
  • Summary
• 7. GenAI in the Lakehouse: Foundations and Architecture
  • From Predictive to Generative Intelligence
    • GenAI Architecture
      • Variational autoencoders
      • Generative adversarial networks
      • Transformers
      • Diffusion models
      • Architecture summary
  • The Mathematics of Scale: Why Bigger Models Work Better
    • Compute-Optimal Training: The Chinchilla Insight
    • Practical Implications for the Lakehouse
  • The Transformer Architecture
    • Why Transformers Succeeded: Parallelism at Scale
    • Three Transformer Variants
    • Core Components of a Transformer Block
    • Autoregressive Generation: From Parallel Training to Sequential Output
      • The generation process
      • Why generation is sequential despite parallel training
    • Implications for the Lakehouse
      • Memory requirements scale with parameters
      • Compute costs dominated by attention
      • Throughput limited by sequential generation
    • Model Serving Optimization in Databricks
    • Unity Catalog Integration: Governance for Foundation Models
    • Infrastructure Trade-Offs and RAG
  • Tokenization, Embeddings, and Context Windows
    • Tokenization: Breaking Text into Learnable Units
    • Embeddings: From Tokens to Geometry
    • Context Windows: The Memory Bottleneck
  • From Probabilities to Text: Decoding Strategies
  • Foundation Models as the New Transfer Learning Paradigm
    • Emergent Capabilities at Scale
    • From Foundation Models to Production Systems
      • Instruction tuning
      • Reinforcement learning from human feedback: aligning with human preferences
      • Parameter-efficient fine-tuning (PEFT): adapting at scale
    • Foundation Models in the Lakehouse
  • GenAI in the Lakehouse
  • Summary
• 8. GenAI in a Databricks Lakehouse Environment
  • Prompt Engineering
    • Understanding Prompts and Prompt Engineering
    • Anatomy of a High-Quality Prompt
    • Prompt Engineering Techniques
      • Zero-shot prompting
      • Few-shot prompting
      • Prompt chaining
      • Chain-of-thought prompting
      • ReAct prompting
    • Practical Tips for Effective Prompting
  • The AI Playground in Databricks
    • Exploring the Playground
    • Model Exploration and Prompt Design
  • Retrieval-Augmented Generation
    • RAG Definition
    • How Do Language Models Learn Knowledge?
      • Model pretraining
      • Model fine-tuning
      • Traditional RAG
    • Factual Recall with Context
      • Why factual recall matters
      • Passing context as open notes
      • Retrieval trade-offs and their mitigation
      • RAG use cases
      • The RAG workflow
    • Vector Stores
      • Core principles
      • Role of vector stores in RAG
      • Integration of vector stores with lakehouse workflows
      • Best practices and architectural considerations
    • Databricks AI Vector Store
  • Practical RAG Implementation
    • Preparing Data for RAG
      • Provision the Unity Catalog
      • Ingesting a subset of Wikipedia
      • Clean and chunk the articles
      • Invoking our cleaning/chunking function
      • Calculating and storing the embeddings
      • Creating our embedding UDF
      • Storing the embeddings in a Delta file
    • Executing Vector Retrieval via Databricks SQL or PySpark
      • Initialization
      • Creating the Vector Search endpoint
      • Creating a Vector Search index
      • Creating a search query and calculating its embedding vector
      • Performing a search
      • Results reranking
      • Running the final query
  • Summary
• 9. AI Agents in the Lakehouse
  • Fundamentals of AI Agents
    • What Is an AI Agent?
    • Benefits of Agents
    • Choosing the Right Execution Model
    • Core Components of a Lakehouse Agent
    • A Lakehouse Agent Reference Architecture
    • Agent Lifecycle
  • Memory Approaches for Lakehouse Agents
    • Agent Memory Shapes
    • Memory Design Patterns
  • Agent Patterns
    • Reflection
    • Tool Use
    • ReAct
    • Planning
    • Multiagent Collaboration
    • Bringing the Patterns Together
  • Failure Modes and Controls for Lakehouse Agents
  • Evidence-Based Diagnostic Analytics with ReAct
    • The Scenario: Asking the Agent Why?
    • Overview of the Lakehouse Architecture
    • ReAct Logic and a Custom-Built Toolkit
    • Code Walkthrough
      • Notebook initialization
      • Set up the Lakehouse environment
      • Define the agent tools
        • DatabricksQueryTool
        • SchemaInfoTool
        • TrendCalculatorTool
      • The ReAct agent implementation
        • The LLM
        • The tool registry
        • The system prompt
        • Conversation memory
        • The analyze method
      • Agent invocation
        • ReAct trace helper
        • Question 1: Diagnose the engagement drop
        • Question 2: Identify the lowest-engagement premium customer
        • Question 3: Memory follow-up
    • ReAct Agent Summary
  • Summary
• 10. The Model Context Protocol
  • What Is MCP?
  • LLM Tool Standardization with MCP
  • REST Versus MCP: Static Contracts Versus Dynamic Discovery
    • The REST Model: Static Contracts
    • The MCP Model: Dynamic Discovery
    • Key Differences
  • MCP Architecture
    • Core Components
    • The Three Primary Server Primitives
    • Why This Architecture Matters in the Lakehouse
  • Deployment Modes and Runtime Behavior
    • Local Mode
    • Remote Mode
    • Scalability and Flexibility
  • Client/Server Communication
    • Phase 1: Initialization and Capability Negotiation
    • Phase 2: Discovery and Invocation
    • Example: Weather API Service
    • Progress Notifications and Long-Running Operations
    • Why the Lifecycle Matters
  • Databricks Unity Catalog MCP Server
    • Key Capabilities
    • Server Architecture and Initialization
    • Run It Now!
      • Step 1: Configure your environment
      • Step 2: Start the server
      • Step 3: Run the Examples
    • Main Solution Components
      • Main entry point
      • Python MCP client
    • Project Structure
    • Design Principles
    • Implementation Walkthrough
      • Implementing MCP tools
        • Tool summary
        • The ToolHandler class
      • Exposing Unity Catalog resources
      • The role of resources in MCP
      • Reusable prompt templates
    • Testing and Interacting with the MCP Server
      • Testing the server with MCP Inspector
        • Invoking a tool
        • Reading a resource
        • Expanding a prompt template
        • Using the Inspector during development
      • Testing in VS Code GitHub Copilot
  • Summary
• 11. Agent-to-Agent Communication and the DSPy Framework
  • A2A Protocol
    • Overview and Foundation
    • Core Architecture
      • Agent cards
      • Agent card versioning
      • Agent skills
      • Task lifecycle management
      • Artifacts
    • Integration with Data Lakehouses
      • Data discovery agents
      • Query optimization agents
      • Multistage pipeline orchestrators
      • Real-time event processing
    • Relationship with MCP
    • Enterprise Adoption and Ecosystem
  • Declarative Self-Improving Python
    • Core Concepts
      • Signatures
      • Modules
      • Teleprompters (optimizers)
      • Metrics
    • DSPy in the Data Lakehouse
      • Data-to-insights pipelines (RAG over the lakehouse)
      • Natural language to SQL
      • Automated data quality
      • Documentation generation
    • DSPy Integration with MLOps and Model Management
    • Advanced DSPy Capabilities for Lakehouse Workloads
  • A2A and DSPy Together: Multiagent Intelligence on the Lakehouse
  • Example: Retail Analytics Assistant on a Databricks Lakehouse
    • Problem Overview
    • Why A2A + DSPy?
    • Architecture Overview
    • Example Interaction
      • Step 1: Planning
      • Step 2: Data discovery
      • Step 3: SQL generation
      • Step 4: Analysis
      • Step 5: Recommendations
      • Step 6: Response synthesis
      • Human-in-the-loop approval
      • Continuous improvement
      • Technology stack summary
    • Running the Example
  • Summary
• Index

pokaż cały spis treści

ukryj recenzje