The Cloud Data Lake Rukmani Gopalan

Spis treści książki

• Preface
  • Why I Wrote This Book
  • Who Should Read This Book?
    • Introducing Klodars Corporation
  • Navigating the Book
  • Conventions Used in This Book
  • OReilly Online Learning
  • How to Contact Us
  • Acknowledgments
• 1. Big DataBeyond the Buzz
  • What Is Big Data?
  • Elastic Data InfrastructureThe Challenge
  • Cloud Computing Fundamentals
    • Cloud Computing Terminology
    • Value Proposition of the Cloud
  • Cloud Data Lake Architecture
    • Limitations of On-Premises Data Warehouse Solutions
    • What Is a Cloud Data Lake Architecture?
    • Benefits of a Cloud Data Lake Architecture
  • Defining Your Cloud Data Lake Journey
  • Summary
• 2. Big Data Architectures on the Cloud
  • Why Klodars Corporation Moves to the Cloud
  • Fundamentals of Cloud Data Lake Architectures
    • A Word on Variety of Data
    • Cloud Data Lake Storage
    • Big Data Analytics Engines
      • MapReduce
      • Apache Hadoop
      • Apache Spark
      • Real-time stream processing pipelines
    • Cloud Data Warehouses
  • Modern Data Warehouse Architecture
    • Reference Architecture
    • Sample Use Case for a Modern Data Warehouse Architecture
    • Benefits and Challenges of Modern Data Warehouse Architecture
  • Data Lakehouse Architecture
    • Reference Architecture for the Data Lakehouse
      • Data formats
      • Metadata
      • Compute engines
    • Sample Use Case for Data Lakehouse Architecture
    • Benefits and Challenges of the Data Lakehouse Architecture
    • Data Warehouses and Unstructured Data
  • Data Mesh
    • Reference Architecture
    • Sample Use Case for a Data Mesh Architecture
    • Challenges and Benefits of a Data Mesh Architecture
  • What Is the Right Architecture for Me?
    • Know Your Customers
    • Know Your Business Drivers
    • Consider Your Growth and Future Scenarios
    • Design Considerations
    • Hybrid Approaches
  • Summary
• 3. Design Considerations for Your Data Lake
  • Setting Up the Cloud Data Lake Infrastructure
    • Identify Your Goals
      • How Klodars Corporation defined the data lake goals
    • Plan Your Architecture and Deliverables
      • How Klodars Corporation planned their architecture and deliverables
    • Implement the Cloud Data Lake
    • Release and Operationalize
  • Organizing Data in Your Data Lake
    • A Day in the Life of Data
    • Data Lake Zones
    • Organization Mechanisms
  • Introduction to Data Governance
    • Actors Involved in Data Governance
    • Data Classification
    • Metadata Management, Data Catalog, and Data Sharing
    • Data Access Management
    • Data Quality and Observability
    • Data Governance at Klodars Corporation
    • Data Governance Wrap-Up
  • Manage Data Lake Costs
    • Demystifying Data Lake Costs on the Cloud
    • Data Lake Cost Strategy
      • Data Lake Environments and Associated Costs
      • Cost strategy based on data
      • Transactions and impact on costs
  • Summary
• 4. Scalable Data Lakes
  • A Sneak Peek into Scalability
    • What Is Scalability?
    • Scale in Our Day-to-Day Life
    • Scalability in Data Lake Architectures
  • Internals of Data Lake Processing Systems
    • Data Copy Internals
      • Components of a data copy solution
      • Understanding resource utilization of a data copy job
    • ELT/ETL Processing Internals
      • Components of an Apache Spark application
      • Understanding resource utilization of a Spark job
    • A Note on Other Interactive Queries
  • Considerations for Scalable Data Lake Solutions
    • Pick the Right Cloud Offerings
      • Hybrid and multicloud solutions
      • IaaS versus PaaS versus SaaS solutions
      • Cloud offerings for Klodars Corporation
    • Plan for Peak Capacity
    • Data Formats and Job Profile
  • Summary
• 5. Optimizing Cloud Data Lake Architectures for Performance
  • Basics of Measuring Performance
    • Goals and Metrics for Performance
    • Measuring Performance
    • Optimizing for Faster Performance
  • Cloud Data Lake Performance
    • SLAs, SLOs, and SLIs
    • Example: How Klodars Corporation Managed Its SLAs, SLOs, and SLIs
  • Drivers of Performance
    • Performance Drivers for a Copy Job
    • Performance Drivers for a Spark Job
  • Optimization Principles and Techniques for Performance Tuning
    • Data Formats
      • Exploring Apache Parquet
      • Other popular data formats
      • How Klodars Corporation picked their data formats
    • Data Organization and Partitioning
      • Optimal data organization strategy for Klodars Corporation
    • Choosing the Right Configurations on Apache Spark
  • Minimize Overheads with Data Transfer
  • Premium Offerings and Performance
    • The Case of Bigger Virtual Machines
    • The Case of Flash Storage
  • Summary
• 6. Deep Dive on Data Formats
  • Why Do We Need These Open Data Formats?
    • Why Do We Need to Store Tabular Data?
    • Why Is It a Problem to Store Tabular Data in a Cloud Data Lake Storage?
  • Delta Lake
    • Why Was Delta Lake Founded?
      • Eliminate data silos across business analysts, data scientists, and data engineers
      • Provide a unified data and computational system for batch and real-time streaming data
      • Support bulk updates or changes to existing data
      • Handle errors due to schema changes and incorrect data
    • How Does Delta Lake Work?
    • When Do You Use Delta Lake?
  • Apache Iceberg
    • Why Was Apache Iceberg Founded?
    • How Does Apache Iceberg Work?
    • When Do You Use Apache Iceberg?
  • Apache Hudi
    • Why Was Apache Hudi Founded?
    • How Does Apache Hudi Work?
      • Copy-on-write tables
      • Merge-on-read tables
    • When Do You Use Apache Hudi?
  • Summary
• 7. Decision Framework for Your Architecture
  • Cloud Data Lake Assessment
    • Cloud Data Lake Assessment Questionnaire
  • Analysis for Your Cloud Data Lake Assessment
    • Starting from Scratch
    • Migrating an Existing Data Lake or Data Warehouse to the Cloud
    • Improving an Existing Cloud Data Lake
  • Phase 1 of Decision Framework: Assess
    • Understand Customer Requirements
    • Understand Opportunities for Improvement
    • Know Your Business Drivers
    • Complete the Assess Phase by Prioritizing the Requirements
  • Phase 2 of Decision Framework: Define
    • Finalize the Design Choices for the Cloud Data Lake
      • Picking your architecture
      • Picking your cloud provider
      • Decision points for data lake migrations
    • Plan Your Cloud Data Lake Project Deliverables
  • Phase 3 of Decision Framework: Implement
  • Phase 4 of Decision Framework: Operationalize
  • Summary
• 8. Six Lessons for a Data Informed Future
  • Lesson 1: Focus on the How and When, Not the If and Why, When It Comes to Cloud Data Lakes
  • Lesson 2: With Great Power Comes Great ResponsibilityData Is No Exception
  • Lesson 3: Customers Lead Technology, Not the Other Way Around
  • Lesson 4: Change Is Inevitable, so Be Prepared
  • Lesson 5: Build Empathy and Prioritize Ruthlessly
  • Lesson 6: Big Impact Does Not Happen Overnight
  • Summary
• A. Cloud Data Lake Decision Framework Template
  • Phase 1: Assess Framework
  • Phase 2: Define Framework
    • Planning the Cloud Data Lake Deliverables
  • Phase 3: Implement Framework
• Index

pokaż cały spis treści

ukryj recenzje