Learning Spark. 2nd Edition (ebook)(audiobook)(audiobook)

Spis treści

• Foreword
• Preface
  • Who This Book Is For
  • How the Book Is Organized
  • How to Use the Code Examples
  • Software and Configuration Used
  • Conventions Used in This Book
  • Using Code Examples
  • OReilly Online Learning
  • How to Contact Us
  • Acknowledgments
• 1. Introduction to Apache Spark: A Unified Analytics Engine
  • The Genesis of Spark
    • Big Data and Distributed Computing at Google
    • Hadoop at Yahoo!
    • Sparks Early Years at AMPLab
  • What Is Apache Spark?
    • Speed
    • Ease of Use
    • Modularity
    • Extensibility
  • Unified Analytics
    • Apache Spark Components as a Unified Stack
      • Spark SQL
      • Spark MLlib
      • Spark Structured Streaming
      • GraphX
    • Apache Sparks Distributed Execution
      • Spark driver
      • SparkSession
      • Cluster manager
      • Spark executor
      • Deployment modes
      • Distributed data and partitions
  • The Developers Experience
    • Who Uses Spark, and for What?
      • Data science tasks
      • Data engineering tasks
      • Popular Spark use cases
    • Community Adoption and Expansion
• 2. Downloading Apache Spark and Getting Started
  • Step 1: Downloading Apache Spark
    • Sparks Directories and Files
  • Step 2: Using the Scala or PySpark Shell
    • Using the Local Machine
  • Step 3: Understanding Spark Application Concepts
    • Spark Application and SparkSession
    • Spark Jobs
    • Spark Stages
    • Spark Tasks
  • Transformations, Actions, and Lazy Evaluation
    • Narrow and Wide Transformations
  • The Spark UI
  • Your First Standalone Application
    • Counting M&Ms for the Cookie Monster
    • Building Standalone Applications in Scala
  • Summary
• 3. Apache Sparks Structured APIs
  • Spark: Whats Underneath an RDD?
  • Structuring Spark
    • Key Merits and Benefits
  • The DataFrame API
    • Sparks Basic Data Types
    • Sparks Structured and Complex Data Types
    • Schemas and Creating DataFrames
      • Two ways to define a schema
    • Columns and Expressions
    • Rows
    • Common DataFrame Operations
      • Using DataFrameReader and DataFrameWriter
        • Saving a DataFrame as a Parquet file or SQL table
      • Transformations and actions
        • Projections and filters
        • Renaming, adding, and dropping columns
        • Aggregations
        • Other common DataFrame operations
    • End-to-End DataFrame Example
  • The Dataset API
    • Typed Objects, Untyped Objects, and Generic Rows
    • Creating Datasets
      • Scala: Case classes
    • Dataset Operations
    • End-to-End Dataset Example
  • DataFrames Versus Datasets
    • When to Use RDDs
  • Spark SQL and the Underlying Engine
    • The Catalyst Optimizer
      • Phase 1: Analysis
      • Phase 2: Logical optimization
      • Phase 3: Physical planning
      • Phase 4: Code generation
  • Summary
• 4. Spark SQL and DataFrames: Introduction to Built-in Data Sources
  • Using Spark SQL in Spark Applications
    • Basic Query Examples
  • SQL Tables and Views
    • Managed Versus UnmanagedTables
    • Creating SQL Databases and Tables
      • Creating a managed table
      • Creating an unmanaged table
    • Creating Views
      • Temporary views versus global temporary views
    • Viewing the Metadata
    • Caching SQL Tables
    • Reading Tables into DataFrames
  • Data Sources for DataFrames and SQL Tables
    • DataFrameReader
    • DataFrameWriter
    • Parquet
      • Reading Parquet files into a DataFrame
      • Reading Parquet files into a Spark SQL table
      • Writing DataFrames to Parquet files
      • Writing DataFrames to Spark SQL tables
    • JSON
      • Reading a JSON file into a DataFrame
      • Reading a JSON file into a Spark SQL table
      • Writing DataFrames to JSON files
      • JSON data source options
    • CSV
      • Reading a CSV file into a DataFrame
      • Reading a CSV file into a Spark SQL table
      • Writing DataFrames to CSV files
      • CSV data source options
    • Avro
      • Reading an Avro file into a DataFrame
      • Reading an Avro file into a Spark SQL table
      • Writing DataFrames to Avro files
      • Avro data source options
    • ORC
      • Reading an ORC file into a DataFrame
      • Reading an ORC file into a Spark SQL table
      • Writing DataFrames to ORC files
    • Images
      • Reading an image file into a DataFrame
    • Binary Files
      • Reading a binary file into a DataFrame
  • Summary
• 5. Spark SQL and DataFrames: Interacting with External Data Sources
  • Spark SQL and Apache Hive
    • User-Defined Functions
      • Spark SQL UDFs
      • Evaluation order and null checking in Spark SQL
      • Speeding up and distributing PySpark UDFs with Pandas UDFs
  • Querying with the Spark SQL Shell, Beeline, and Tableau
    • Using the Spark SQL Shell
      • Create a table
      • Insert data into the table
      • Running a Spark SQL query
    • Working with Beeline
      • Start the Thrift server
      • Connect to the Thrift server via Beeline
      • Execute a Spark SQL query with Beeline
      • Stop the Thrift server
    • Working with Tableau
      • Start the Thrift server
      • Start Tableau
      • Stop the Thrift server
  • External Data Sources
    • JDBC and SQL Databases
      • The importance of partitioning
    • PostgreSQL
    • MySQL
    • Azure Cosmos DB
    • MS SQL Server
    • Other External Sources
  • Higher-Order Functions in DataFrames and Spark SQL
    • Option 1: Explode and Collect
    • Option 2: User-Defined Function
    • Built-in Functions for Complex Data Types
    • Higher-Order Functions
      • transform()
      • filter()
      • exists()
      • reduce()
  • Common DataFrames and Spark SQL Operations
    • Unions
    • Joins
    • Windowing
    • Modifications
      • Adding new columns
      • Dropping columns
      • Renaming columns
      • Pivoting
  • Summary
• 6. Spark SQL and Datasets
  • Single API for Java and Scala
    • Scala Case Classes and JavaBeans for Datasets
  • Working with Datasets
    • Creating Sample Data
    • Transforming Sample Data
      • Higher-order functions and functional programming
      • Converting DataFrames to Datasets
  • Memory Management for Datasets and DataFrames
  • Dataset Encoders
    • Sparks Internal Format Versus Java Object Format
    • Serialization and Deserialization (SerDe)
  • Costs of Using Datasets
    • Strategies to Mitigate Costs
  • Summary
• 7. Optimizing and Tuning Spark Applications
  • Optimizing and Tuning Spark for Efficiency
    • Viewing and Setting Apache Spark Configurations
    • Scaling Spark for Large Workloads
      • Static versus dynamic resource allocation
      • Configuring Spark executors memory and the shuffle service
      • Maximizing Spark parallelism
        • How partitions are created
  • Caching and Persistence of Data
    • DataFrame.cache()
    • DataFrame.persist()
    • When to Cache and Persist
    • When Not to Cache and Persist
  • A Family of Spark Joins
    • Broadcast Hash Join
      • When to use a broadcast hash join
    • Shuffle Sort Merge Join
      • Optimizing the shuffle sort merge join
      • When to use a shuffle sort merge join
  • Inspecting the Spark UI
    • Journey Through the Spark UI Tabs
      • Jobs and Stages
      • Executors
      • Storage
      • SQL
      • Environment
      • Debugging Spark applications
  • Summary
• 8. Structured Streaming
  • Evolution of the Apache Spark Stream Processing Engine
    • The Advent of Micro-Batch Stream Processing
    • Lessons Learned from Spark Streaming (DStreams)
    • The Philosophy of Structured Streaming
  • The Programming Model of Structured Streaming
  • The Fundamentals of a Structured Streaming Query
    • Five Steps to Define a Streaming Query
      • Step 1: Define input sources
      • Step 2: Transform data
      • Step 3: Define output sink and output mode
      • Step 4: Specify processing details
      • Step 5: Start the query
      • Putting it all together
    • Under the Hood of an Active Streaming Query
    • Recovering from Failures with Exactly-Once Guarantees
    • Monitoring an Active Query
      • Querying current status using StreamingQuery
        • Get current metrics using StreamingQuery
        • Get current status using StreamingQuery.status()
      • Publishing metrics using Dropwizard Metrics
      • Publishing metrics using custom StreamingQueryListeners
  • Streaming Data Sources and Sinks
    • Files
      • Reading from files
      • Writing to files
    • Apache Kafka
      • Reading from Kafka
      • Writing to Kafka
    • Custom Streaming Sources and Sinks
      • Writing to any storage system
        • Using foreachBatch()
        • Using foreach()
      • Reading from any storage system
  • Data Transformations
    • Incremental Execution and Streaming State
    • Stateless Transformations
    • Stateful Transformations
      • Distributed and fault-tolerant state management
      • Types of stateful operations
  • Stateful Streaming Aggregations
    • Aggregations Not Based on Time
    • Aggregations with Event-Time Windows
      • Handling late data with watermarks
        • Semantic guarantees with watermarks
      • Supported output modes
  • Streaming Joins
    • StreamStatic Joins
    • StreamStream Joins
      • Inner joins with optional watermarking
      • Outer joins with watermarking
  • Arbitrary Stateful Computations
    • Modeling Arbitrary Stateful Operations with mapGroupsWithState()
    • Using Timeouts to Manage Inactive Groups
      • Processing-time timeouts
      • Event-time timeouts
    • Generalization with flatMapGroupsWithState()
  • Performance Tuning
  • Summary
• 9. Building Reliable Data Lakes with Apache Spark
  • The Importance of an Optimal Storage Solution
  • Databases
    • A Brief Introduction to Databases
    • Reading from and Writing to Databases Using Apache Spark
    • Limitations of Databases
  • Data Lakes
    • A Brief Introduction to Data Lakes
    • Reading from and Writing to Data Lakes using Apache Spark
    • Limitations of Data Lakes
  • Lakehouses: The Next Step in the Evolution of Storage Solutions
    • Apache Hudi
    • Apache Iceberg
    • Delta Lake
  • Building Lakehouses with Apache Spark and Delta Lake
    • Configuring Apache Spark with Delta Lake
    • Loading Data into a Delta Lake Table
    • Loading Data Streams into a Delta Lake Table
    • Enforcing Schema on Write to Prevent Data Corruption
    • Evolving Schemas to Accommodate Changing Data
    • Transforming Existing Data
      • Updating data to fix errors
      • Deleting user-related data
      • Upserting change data to a table using merge()
      • Deduplicating data while inserting using insert-only merge
    • Auditing Data Changes with Operation History
    • Querying Previous Snapshots of a Table with Time Travel
  • Summary
• 10. Machine Learning with MLlib
  • What Is Machine Learning?
    • Supervised Learning
    • Unsupervised Learning
    • Why Spark for Machine Learning?
  • Designing Machine Learning Pipelines
    • Data Ingestion and Exploration
    • Creating Training and Test Data Sets
    • Preparing Features with Transformers
    • Understanding Linear Regression
    • Using Estimators to Build Models
    • Creating a Pipeline
      • One-hot encoding
    • Evaluating Models
      • RMSE
        • Interpreting the value of RMSE
        • R2
    • Saving and Loading Models
  • Hyperparameter Tuning
    • Tree-Based Models
      • Decision trees
      • Random forests
    • k-Fold Cross-Validation
    • Optimizing Pipelines
  • Summary
• 11. Managing, Deploying, and Scaling Machine Learning Pipelines with Apache Spark
  • Model Management
    • MLflow
      • Tracking
  • Model Deployment Options with MLlib
    • Batch
    • Streaming
    • Model Export Patterns for Real-Time Inference
  • Leveraging Spark for Non-MLlib Models
    • Pandas UDFs
    • Spark for Distributed Hyperparameter Tuning
      • Joblib
      • Hyperopt
  • Summary
• 12. Epilogue: Apache Spark 3.0
  • Spark Core and Spark SQL
    • Dynamic Partition Pruning
    • Adaptive Query Execution
      • The AQE framework
    • SQL Join Hints
      • Shuffle sort merge join (SMJ)
      • Broadcast hash join (BHJ)
      • Shuffle hash join (SHJ)
      • Shuffle-and-replicate nested loop join (SNLJ)
    • Catalog Plugin API and DataSourceV2
    • Accelerator-Aware Scheduler
  • Structured Streaming
  • PySpark, Pandas UDFs, and Pandas Function APIs
    • Redesigned Pandas UDFs with Python Type Hints
    • Iterator Support in Pandas UDFs
    • New Pandas Function APIs
  • Changed Functionality
    • Languages Supported and Deprecated
    • Changes to the DataFrame and Dataset APIs
    • DataFrame and SQL Explain Commands
  • Summary
• Index

pokaż cały spis treści

ukryj recenzje